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1.0  INTRODUCTION 


This  is  one  of  a  series  of  the  Environmental  Security  Technology  Certification  Program  (ESTCP) 
live  site  demonstrations  of  classification  technologies  for  Munitions  Response  (MR).  In  this 
demonstration,  Arcadis  United  States  (U.S.),  Incorporate  (Arcadis)  performed  advanced 
geophysical  classification  (AGC)  of  cued  Time-domain  Electromagnetic  Multi-Sensor  Towed 
Array  Detection  System  2x2  (TEMTADS)  data  that  was  collected  by  URS  Group,  Inc.  (URS)  at 
the  Castner  Range  (CR)  at  Fort  Bliss,  Texas. 

1.1  BACKGROUND 

ESTCP  contracted  Arcadis  to  conduct  AGC  analysis  of  the  TEMTADS  data  that  was  collected  by 
URS  at  the  Closed  Castner  Range  as  part  of  AGC  live  site  demonstrations  to  transfer  AGC 
technologies  to  Arcadis.  Mr.  Steve  Stacy  (Analyst  1)  and  Mr.  Gabriel  Hebert  (Analyst  2) 
performed  AGC  analysis  of  the  TEMTADS  dataset  separately.  A  list  of  points  of  contact  are 
included  in  Appendix  A. 

1.2  OBJECTIVE  OF  THE  DEMONSTRATION 

Arcadis  has  previously  demonstrated  AGC  approaches  another  ESTCP  live  site  demonstration 
(Arcadis,  2013;  Arcadis  2015a).  The  primary  objectives  of  this  demonstration  were  to  determine 
the  effectiveness  of  the  technology  in  minimizing  the  number  of  anomalies  that  require  intrusive 
investigation  while  still  recovering  all  the  targets  of  interest  (TOI),  which  include  munitions  and 
explosives  of  concern  (MEC)  and  industry  standard  objects  (ISOs)  used  as  blind  seed  items  (BSIs). 

The  goals  of  the  demonstration  included: 

•  Demonstrate  the  cost  and  performance  of  the  TEMTADS  and  AGC. 

•  Train  Arcadis  analysts  on  the  application  of  AGC  to  a  challenging  site  with  varying 
background  response. 

1.3  REGULATORY  DRIVERS 

The  Department  of  Defense’s  (DoD’s)  Military  Munitions  Response  Program  (MMRP)  is  charged 
with  characterizing  and,  where  necessary,  remediating  munitions  response  sites  (MRSs).  When 
an  MRS  is  remediated,  it  is  typically  mapped  with  a  geophysical  system,  (i.e.,  either  a 
magnetometer  or  electromagnetic  induction  [EMI]  sensor),  and  the  locations  of  all  detectable 
signals  are  excavated.  Many  of  these  detections  do  not  correspond  to  munitions,  but  rather  to  other 
harmless  metallic  objects  or  geology:  field  experience  indicates  that  often  in  excess  of  90%  of 
objects  excavated  during  the  course  of  a  MR  are  found  to  be  nonhazardous  items.  Current 
geophysical  technology,  as  it  is  traditionally  implemented,  does  not  provide  a  physics-based, 
quantitative,  validated  means  to  discriminate  between  hazardous  munitions  and  nonhazardous 
scrap  metal. 
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With  no  infonnation  to  suggest  the  origin  of  the  signals,  all  anomalies  are  currently  treated  as 
though  they  are  intact  munitions  when  they  are  dug.  They  are  carefully  excavated  by  unexploded 
ordnance  (UXO)  technicians  using  a  process  that  often  requires  expensive  safety  measures,  such 
as  barriers  or  exclusion  zones.  Thus,  most  of  the  costs  to  remediate  a  munitions-impacted  site  are 
currently  spent  on  excavating  targets  that  pose  no  threat.  If  these  items  could  be  determined  with 
high  confidence  to  be  nonhazardous,  some  of  these  expensive  measures  could  be  eliminated  or  the 
items  could  be  left  unexcavated  entirely. 

The  MMRP  is  severely  constrained  by  available  resources.  Remediation  of  the  entire  inventory 
using  current  practices  is  cost  prohibitive  within  current  and  anticipated  funding  levels.  With 
current  planning,  estimated  MR  completion  dates  on  many  sites  are  decades  out.  The  Office  of 
the  Under  Secretary  of  Defense  for  Acquisition  (OSDA)  observed  in  its  2003  report  that  significant 
cost  savings  could  be  realized  if  successful  classification  between  munitions  and  other  sources  of 
anomalies  could  be  implemented  (OSDA,  2003).  If  these  savings  were  realized,  the  limited 
resources  of  the  MMRP  could  be  used  to  accelerate  the  remediation  of  MRSs  that  are  currently 
forecast  to  be  untouched  for  decades. 
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2.0  TECHNOLOGY 


This  demonstration  consisted  of  AGC  analysis  of  cued  TEMTADS  data.  Details  of  this  technology 
and  a  brief  description  of  the  major  components  of  the  demonstration  are  provided  below. 

2.1  TECHNOLOGY  DESCRIPTION 

2.1.1  TEMTADS 

The  TEMTADS  is  a  man  portable,  advanced  EMI  sensor  array  (see  Figure  2-1)  developed  by  the 
Naval  Research  Laboratory  (NRL)  and  based  on  their  larger,  5x5  TEMTADS  array.  The 
TEMTADS  consists  of  four  35-centimeter  (cm)  transmit  coils  with  four  8  cm  tri-axial  receiver 
cubes  (see  Figure  2-2).  The  TEMTADS  was  developed  through  ESTCP  (Kingdon,  2012)  and  has 
been  shown  to  reliably  retain  the  performance  of  the  original  TEMTADS  in  a  much  smaller  size, 
which  enables  the  man  portable  version  to  access  difficult  terrain  where  mobility  is  limited 
(ESTCP,  2012a  and  2012b;  Kingdon,  2012).  URS  operated  the  TEMTADS  system  in  static 
classification  mode  at  CR  in  a  litter  mode  (not  pictured). 

Figure  2-1:  TEMTADS  2x2 
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Figure  2-2:  TEMTADS  Coil  Geometry 


+  Y 

z 

t 


+x 


m 


Sensor  Orientation 

Direction  of  Travel 


EM  Sensor 


2.2  TECHNOLOGY  DEVELOPMENT 

Technologies  used  during  this  demonstration  include  the  above  TEMTADS  sensor  with  real  time 
kinematic  (RTK)  differential  global  positioning  system  (DGPS),  and  the  UX-Analyze  Advanced 
module  within  Geosoft  Oasis  Montaj®.  The  TEMTADS  and  UX-Analyze  Advanced  were 
developed  under  ESTCP  and  further  descriptions  of  their  development  can  be  found  in  the 
following  reports: 

•  TEMTADS:  Kingdon  et.  al,  2012; 

•  UX-Analyze  Advanced:  Keiswetter,  2009,  and  the  following  projects,  which  do  not 
currently  have  technical  reports  that  are  available: 

o  MR-201 164:  Demonstration  of  Physics-Inspired  Classification  Methodologies  for 
MR 

o  MR-2013 12:  UXO  Classification  Demonstrations  at  Live  Sites  Using  UX-Analyze 

2.3  ADVANTAGES  AND  LIMITATIONS  OF  THE  TECHNOLOGY 

The  major  advantage  of  the  TEMTADS  and  UX-Analyze  Advanced  module  is  that  combined,  they 
provide  the  ability  to  classify  anomalies  as  being  due  to  either  TOI  or  non-TOI.  This  can  lead  to 
significant  cost  savings  in  MR  cleanups.  Conventional  digital  geophysical  mapping  (DGM) 
sensors  ( e.g .,  EM61-MK2)  have  very  limited  ability  to  correctly  classify  TOI  and  non-TOI.  Other 
advanced  EMI  sensors  have  also  been  successful  in  ESTCP  funded  classification  demonstrations; 
however,  they  were  not  used  during  this  live  site  demonstration. 
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3.0  PERFORMANCE  OBJECTIVES 


Performance  objectives  or  measurement  quality  objectives  (MQOs)  were  developed  and  document 
in  the  project’s  Uniform  Federal  Policy-  Quality  Assurance  Project  Plan  (UFP-QAPP)  to  measure 
the  quality  of  the  data  collected  during  the  ESTCP  live  site  demonstration.  MQOs  are  outlined 
below. 


3.1  STATIC  DATA  PERFORMANCE  OBJECTIVES 

The  perfonnance  objectives  for  the  cued  TEMTADS  data  collection  and  interpretation  are 
summarized  in  Table  3-1  and  the  below  sections. 

Table  3-1:  Cued  TEMTADS  Data  Collection  and  Analysis  Objectives 


Performance 

Objective 

Metric 

Data  Required 

Success  Criteria 

Confirm  all 
background 
measurements  are 
valid 

Sensor  response 

Background 

measurements 

All  decay  amplitudes  lower 
than  project  threshold  and 
qualitatively  agree  with  initial 
measurement 

Confirm  all 

measurements 
have  an  applicable 
background 

Background 

measurements 

Cued  survey  data 

Time  Separation  between 
background  measurement  and 
anomaly  measurement  <  2 
hours  (hrs) 

Valid  Position 
Data  (1):  GPS 

GPS  status  flag 

Cued  survey  data 

Confirm  GPS  status  flag 
indicates  RTK  fix 

Valid  Position 
Data  (2):  Inertial 
Measurement 
Unit  (IMU) 

IMU  Orientation 
data 

Cued  survey  data 

Orientation  data  valid  data 
input  string  checksum  passes 

Confirm  inversion 
model  supports 
classification  (1  of 

3) 

Fit  Coherence 

Cued  survey  data 

Derived  model  response  must 
fit  the  observed  data  with  a  fit 
coherence  >0.8 

Confirm  inversion 
model  supports 
classification  (2  of 

3) 

Fit  location 

Cued  survey  data 

Fit  location  estimate  of  item  < 
0.4-meters  (m)  from  center  of 
sensor 

Confirm  inversion 
model  supports 
classification  (3  of 

3) 

Fit  Location 

Cued  survey  data 

100%  of  blind  seed  item 
positions  <  0.25-m  from 
predicted  position  (x,  y,  z) 

Confirm  derived 
features  match 

Fit  Location 

Prioritized 
classification  dig  list 
and  dig  results 

100%  of  recovered  (excluding 
can’t  analyze  category)  item 
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Performance 

Objective 

Metric 

Data  Required 

Success  Criteria 

ground  truth  (1  of 
2) 

positions  <  0.25-m  from 
predicted  position  (x,  y) 

Confirm  derived 
features  match 
ground  truth  (2  of 
2) 

Classification 

results 

Prioritized 
classification  dig  list 
and  dig  results 

100%  of  recovered  object  size 
estimates  (excluding  can’t 
analyze  category) 
qualitatively  match  predicted 
size 

Validation  of 
TOI/non-TOI 
thresholds 

Classification 

results 

Prioritized 
classification  dig  list 
and  dig  results 

100%  of  predicted  non-TOI 
intrusively  investigated  are 
non-TOI 

3.1.1  Objective:  Confirm  all  background  measurements  are  valid 

The  reliability  of  the  production  data  required  background  measurements  that  were  free  of  metallic 
objects  that  could  adversely  affect  background  corrections.  Ongoing  background  measurements 
were  required  to  minimize  time-variable  changes  in  background  response. 

3. 1.1.1.  Metric 

The  metric  for  this  objective  were  the  background  measurement  responses. 

3. 1.1.2.  Data  Requirements 

The  initial  and  ongoing  cued  background  measurements  were  used  to  evaluate  success  of  this 
objective. 

3. 1.1.3.  Success  Criteria 

The  objective  was  met  if  decay  amplitudes  qualitatively  agreed  with  the  initial  background 
measurement. 

3.1.2  Objective:  Confirm  all  measurements  have  an  applicable  background 

The  reliability  of  the  production  data  required  background  measurements  be  collected  to  record 
time-variable  effects  in  background  response  to  minimize  their  impact  to  the  classification  results. 

3. 1.2.1.  Metric 

The  metric  for  this  objective  were  the  background  measurement  responses. 

3. 1.2.2.  Data  Requirements 

Background  measurements  were  nominally  collected  every  2-hrs. 
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3. 1.2.3.  Success  Criteria 


The  objective  was  met  if  the  time  separation  between  background  measurement  and  anomaly 
measurement  is  less  than  2-hrs. 

3.1.3  Objective:  Valid  Position  Data  -  GPS 

Data  acquisition  software  was  monitored  to  ensure  that  all  data  streams  ( e.g .,  GPS,  IMU)  were 
valid  and  being  recorded.  The  objective  was  to  have  valid  position  data  from  the  GPS. 

3. 1.3.1.  Metric 

The  GPS  status  flag  and  GPS  quality  metric  were  used  to  evaluate  performance. 

3. 1.3.2.  Data  Requirements 

The  GPS  position  of  the  cued  survey  data  was  used  to  evaluate  success  of  this  objective. 

3. 1.3.3.  Success  Criteria 

Successful  completion  required  that  the  GPS  status  flag  indicated  RTK  fix. 

3.1.4  Objective:  Valid  Position  Data  -  IMU 

Data  acquisition  software  was  monitored  to  ensure  that  all  data  streams  (e.g.,  GPS,  IMU)  were 
valid  and  being  recorded.  The  objective  was  to  have  valid  position  data  from  the  IMU. 

3. 1.4.1.  Metric 

The  IMU  status  was  used  to  evaluate  success  of  this  objective. 

3. 1.4.2.  Data  Requirements 

The  cued  TEMTADS  data  was  used  to  evaluate  perfonnance  under  this  objective. 

3. 1.4.3.  Success  Criteria 

Successful  completion  required  a  valid  orientation  on  the  data. 

3.1.5  Objective:  Confirm  Inversion  Model  Supports  Classification  (1  of  3) 

The  reliability  of  the  classification  results  depends  on  collecting  data  that  had  a  relatively  high  fit 
coherence  to  indicate  the  data  can  be  modeled. 

3. 1.5.1. Metric 

The  metric  for  this  objective  was  the  fit  coherence. 
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3. 1.5.2.  Data  Requirements 

The  modeled  and  detected  cued  survey  data  was  used  to  evaluate  success  of  this  objective. 

3. 1.5.3.  Success  Criteria 

Successful  completion  required  that  the  derived  model  response  must  fit  the  observed  data  with  a 
fit  coherence  of  greater  than  or  equal  to  0.8. 

3.1.6  Objective:  Confirm  Inversion  Model  Supports  Classification  (2  of  3) 

The  reliability  of  the  classification  results  depended  on  collecting  data  where  the  sensor  is  above 
the  subsurface  metallic  object. 

3. 1.6.1.  Metric 

The  metrics  for  this  objective  were  the  sensor  array  and  fit  location. 

3. 1.6.2.  Data  Requirements 

The  cued  survey  data  and  classification  fit  locations  were  used  to  evaluate  success  of  this  objective. 

3. 1.6.3.  Success  Criteria 

The  objective  was  met  if  the  fit  location  estimate  of  the  subsurface  item  was  less  than  or  equal  to 
0.4-m  from  center  of  sensor. 

3.1.7  Objective:  Confirm  derived  features  match  ground  truth  (1  of  2) 

The  reliability  of  the  classification  results  depended  on  reliably  being  able  to  classify  targets  at 
their  respective  location. 

3. 1.7.1.  Metric 

The  metrics  for  this  objective  were  the  fit  location  and  the  known  BSI  location. 

3. 1.7.2.  Data  Requirements 

The  cued  survey  data  were  used  to  evaluate  success  of  this  objective. 

3. 1.7.3.  Success  Criteria 

The  objective  was  met  if  100%  of  predicted  seed  positions  was  less  than  or  equal  to  0.25  m  from 
known  position  (x,  y,  z). 
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3.1.8  Objective:  Confirm  derived  features  match  ground  truth  (2  of  2) 

The  reliability  of  the  classification  results  depended  on  reliably  being  able  to  classify  the  size  of 
targets. 

3. 1.8.1.  Metric 

The  metric  for  this  objective  were  the  classification  and  results. 

3. 1.8.2.  Data  Requirements 

The  prioritized  classification  dig  list  and  dig  results  were  used  to  evaluate  success  of  this  objective. 

3. 1.8.3.  Success  Criteria 

The  objective  was  met  if  100%  of  recovered  object  size  estimates  (excluding  can’t  analyze 
category)  qualitatively  matched  predicted  size. 

3.1.9  Objective:  Validation  of  TOI/non-TOI  thresholds 

The  data  reliability  depended  on  the  classification  approach  to  correctly  identify  TOI  and  non- 
TOI. 

3. 1.9.1.  Metric 

The  metric  for  this  objective  was  the  classification  results. 

3. 1.9.2.  Data  Requirements 

The  prioritized  classification  dig  list  and  dig  results  were  used  to  evaluate  success  of  this  objective. 

3. 1.9.3.  Success  Criteria 

The  objective  was  met  if  100%  of  predicted  non-TOI  intrusively  investigated  are  non-TOI. 
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4.0  SITE  DESCRIPTION 

See  the  ESTCP  Programmatic  Office’s  demonstration  plan  for  details  on  the  Closed  Castner  Range 
Conceptual  Site  Model. 
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5.0  TEST  DESIGN 


The  objective  of  this  demonstration  was  to  demonstrate  a  methodology  for  the  use  of  AGC  in  the 
MR  process.  The  three  key  components  of  this  methodology  are  collection  of  high-quality 
geophysical  data  and  principled  selection  of  anomalous  regions  in  those  data,  analysis  of  the 
selected  anomalies  using  physics-based  models  to  extract  target  parameters  such  as  size,  shape, 
and  material  properties,  and  the  use  of  those  parameters  to  construct  a  ranked  anomaly  list.  Each 
of  these  components  were  handled  separately  in  this  program.  URS  was  responsible  for  collection 
and  quality  of  the  TEMTADS  data.  Arcadis  was  responsible  for  evaluating  this  data  and 
developing  ranked  anomaly  lists. 

Each  Arcadis  analyst  submitted  a  ranked  anomaly  list  for  the  TEMTADS  data  set.  All  anomalies 
were  categorized  and  placed  on  the  dig  list  in  the  following  order: 

•  Category  -1:  Anomalies  for  which  training  labels  had  been  requested.  These  anomalies 
were  placed  at  the  top  of  the  dig  list. 

•  Category  0:  Anomalies  for  which  Arcadis  was  not  able  to  extract  reliable  parameters. 

•  Category  1:  Anomalies  that  had  a  high  likelihood  of  being  a  TOI  based  on  the  analyst’s 
classification  method. 

•  Category  2:  Anomalies  for  which  the  analysts  were  unsure  whether  the  anomalies  were 
TOI  or  non-TOI.  Category  2  anomalies,  as  well  as  those  listed  above,  were  placed  above 
the  analyst’s  stop  dig  threshold. 

•  Category  3:  Anomalies  that  had  a  high  likelihood  of  being  clutter  items  and/or  a  low 
likelihood  of  being  TOI.  These  anomalies  were  placed  below  the  dig  threshold. 

These  inputs  were  scored  by  the  Institute  for  Defense  Analyses  (IDA)  with  emphasis  on  the 
number  of  items  that  were  correctly  labeled  nonhazardous  ( i.e .,  non-TOI)  while  correctly  labeling 
all  TOI. 

The  primary  objective  of  the  demonstration  was  to  assess  how  well  the  two  Arcadis  analysts  could 
order  our  ranked  anomaly  lists  and  specify  the  threshold  separating  high  confidence  clutter  from 
all  other  items. 

5.1  CONCEPTUAL  EXPERIMENTAL  DESIGN 

Arcadis  performed  the  following  tasks  to  achieve  this  overall  objective: 

•  Data  Analysis 

o  Pre-Processing  and  QC  (See  Section  6.1) 
o  Parameter  estimation  (See  Section  6.3) 
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o  Classification  and  classifier  training  (See  Section  6.4) 
o  Classification  (See  Section  6.5) 

•  Data  Products  (See  Section  6.6) 
o  Ranked  anomaly  list 

5.2  SITE  PREPARATION 

See  the  ESTCP  Programmatic  Office’s  report  for  details  on  the  site  preparation  activities. 

5.3  SYSTEM  SPECIFICATION 
5.3.1  TEMTADS  MP  2x2 

URS  collected  static  TEMTADS  data  at  1,492  of  the  detected  anomalies.  Arcadis’  two  data 
analysts  imported  these  files  into  Geosoft  Oasis  Montaj'  for  processing  using  the  UX-Analyze 
Advanced  module  and  perfonned  QC,  background  corrections,  and  AGC  for  each  of  the 
anomalies. 

See  the  ESTCP  Programmatic  Office’s  report  for  details  on  the  data  acquisition  parameters  that 
were  used  to  collect  cued  TEMTADS  data. 

5.4  CALIBRATION  ACTIVITIES 

See  the  ESTCP  Programmatic  Office’s  report  for  details  on  site  calibration  activities. 

5.5  DATA  COLLECTION 

See  the  ESTCP  Programmatic  Office’s  report  for  details  on  the  data  collection  efforts. 

5.6  VALIDATION 

See  the  ESTCP  Program  Office’s  report  for  details  on  the  data  validation  efforts. 
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6.0  DATA  ANALYSIS  AND  PRODUCTS 


The  two  Arcadis  data  analysts  processed,  and  analyzed  the  cued  TEMTADS  data  by  subtracting 
background  measurements  and  performing  AGC  of  the  data. 

Once  the  static  data  was  pre-processed,  Arcadis  employed  physics-based  models  to  extract  target 
parameters  and  then  applied  a  library  matching  classification  algorithm  contained  in  the  Oasis 
Montaj  advanced  UX-Analyze  Advanced  module  to  produce  a  ranked  anomaly  list. 

6.1  PREPROCESSING 

URS  performed  data  pre-processing.  See  the  ESTCP  Programmatic  Office’s  report  for  further 
details  on  pre-processing. 

6.2  TARGET  SELECTION  FOR  DETECTION 

See  the  ESTCP  Program  Office’s  report  for  details  on  target  selection  for  detection. 

6.3  PARAMETER  ESTIMATES 

Arcadis  analyzed  a  total  of  1,492  cued  TEMTADS.  Each  anomaly  was  analyzed  using  the  single 
and  multi-object  solver  algorithms  in  the  UX-Analyze  Advanced  module  within  Geosoft  Oasis 
Montaj10.  Both  intrinsic  (size,  shape,  materials  properties)  and  extrinsic  (location,  depth, 
orientation)  parameters  were  estimated  in  these  analyses  and  a  list  of  the  relevant  target  parameters 
from  each  analysis  compiled. 

The  static  data  was  processed  using  UX-Analyze  Advanced  module  to  extract  the  three  principal 
axis  polarizability  curves  for  each  target.  Arcadis  then  matched  the  polarizability  curves  for  each 
target  to  a  library  of  polarizability  curves  to  classify  the  target  as  either  TOI  or  non-TOI. 

6.4  CLASSIFIER 

Arcadis’  classifier  involved  matching  the  measured  polarizabilities  to  a  library  that  contained  TOI 
from  previous  sites.  The  size  and  shape  of  polarizabilities  (or  (Is)  were  matched  to  the  known 
library  items  for  the  following  four  scenarios: 

•  3  component  target  classification 

o  Size  -  Pi 
o  Shape  1  —  P2/  pi 
o  Shape  1  -  P3/  Pi 

•  2  component  target  classification  (1) 
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o  Size  -  Pi 
o  Shape  1  —  P2/  pi 


•  2  component  target  classification  (2) 

o  Size  -  Pi 
o  Shape  1  -  p3/  Pi 

•  1  component  target  classification 

o  Size  -  Pi 

Based  on  this  initial  classification,  both  data  analysts  requested  training  data  from  the  program 
office  to  verify  the  library  matching  model  that  was  being  proposed.  The  two  data  analysts 
requested  the  following  training  data: 

•  Analyst  1: 

o  Request  1 :  43  anomalies 
o  Request  2:  7  anomalies 

•  Analyst  2: 

o  Request  1 :  25  anomalies 
o  Request  2:  4  anomalies 

Analyst  1  requested  training  data  to  detennine  whether  small  munitions  (e.g.,  fuzes,  20-mm 
projectiles,  and  37-mm  projectiles)  were  present  on  the  site  and  for  anomalies  that  had  relatively 
low  confidence  metrics  (e.g.,  0.7  -  0.85)  for  munitions  suspected  to  be  present  at  the  site  to  aid  in 
detennining  the  appropriate  location  of  the  dig  threshold.  Of  the  requested  training  data,  only  2 
targets  were  TOIs.  The  remaining  anomalies  selected  confirmed  that  fuzes  and  20-mm  projectiles 
weren’t  anticipated  TOI  at  the  site. 

Analyst  2  requested  calibration  data  to  assist  in  identifying  an  appropriate  dig/no-dig  threshold  and 
to  determine  whether  small  munitions  were  present  on  the  site.  Two  separate  analyst  calibration 
dig  lists  were  submitted.  Within  these  lists,  two  items  were  confirmed  as  TOI.  These  items  were 
both  37-mm  projectiles,  and  were  confirmed  as  high  confidence  matches  in  the  initial  classifier. 
The  remaining  targets  were  primarily  frag  and  small  arms  items.  The  infonnation  from  the 
calibration  digs  were  used  to  refine  the  classification  library  on  the  basis  that  items  smaller  than 
37-mm  projectiles  were  not  likely  to  be  present  at  the  site. 

6.5  CLASSIFICATION 

6.5.1  Initial  Classification 

Analysts  1  and  2  each  produced  a  ranked  anomaly  list  for  the  TEMTADS  library  matching 
approach.  The  library  matching  approach  for  both  sensors  included  matching  to  both  suspected 
TOI  at  the  site,  which  included  small  ISOs,  37-mm  projectiles,  and  larger  munitions.  The  final 
classification  used  a  combined  confidence  metric  that  averaged  the  confidence  metric  for  the  best 
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fitting  TOI  (or  clutter  item  in  the  library)  for  the  3-,  2-,  and  1-  component  matches.  The  best  fitting 
TOI  (or  clutter  item)  for  the  3-component  match  was  then  assigned  as  the  TOI  Type.  Each  analyst 
then  classified  each  anomaly  as  one  of  the  following  categories: 

•  Analyst  1 

o  Category  -1:  anomalies  for  which  training  labels  had  been  requested.  These 
anomalies  were  placed  at  the  top  of  the  dig  list, 
o  Category  0:  Anomalies  for  which  Arcadis  was  not  able  to  extract  reliable 
parameters. 

o  Category  1:  Anomalies  that  had  a  combined  metric  match  to  TOI  greater  than 
0.925.  These  anomalies  were  believed  to  have  a  high  likelihood  of  being  TOI. 
o  Category  2:  These  anomalies  were  anomalies  for  which  Analyst  1  was  unsure 
whether  the  anomaly  was  TOI  or  non-TOI  and  were  classified  as  follows: 

■  Targets  that  met  all  criteria  in  Table  1  in  the  Analyst  1  Anomaly  Decision 
Memo  (See  Appendix  B)  with  decision  statistics  between  0.825  and  0.925, 
and 

■  Non-weak  targets  that  had  decision  statistics  above  0.75. 

o  Category  3:  Anomalies  that  had  a  combined  metric  match  to  TOI  that  didn’t  meet 
any  of  the  above  criteria.  These  anomalies  had  a  high  likelihood  of  being  clutter 
items  and/or  a  low  likelihood  of  being  TOI.  These  anomalies  were  placed  below 
the  dig  threshold. 

•  Analyst  2 

o  Category  -1:  anomalies  for  which  training  labels  had  been  requested.  These 
anomalies  were  placed  at  the  top  of  the  dig  list, 
o  Category  0:  Anomalies  for  which  Arcadis  was  not  able  to  extract  reliable 
parameters. 

o  Category  1:  Anomalies  that  had  a  combined  metric  match  to  TOI  greater  than 
0.925.  These  anomalies  were  believed  to  have  a  high  likelihood  of  being  TOI. 
o  Category  2:  Anomalies  that  had  a  combined  metric  match  to  TOI  between  0.78  and 
0.925.  Arcadis  was  unsure  whether  these  anomalies  were  TOI  or  non-TOI. 
o  Category  3:  Anomalies  that  had  a  combined  metric  match  to  TOI  less  than  0.78. 
These  anomalies  had  a  high  likelihood  of  being  clutter  items  and/or  a  low  likelihood 
of  being  TOI.  These  anomalies  were  placed  below  the  dig  threshold. 

Table  6-1  shows  an  example  of  how  both  Arcadis  data  analysist  ordered  their  ranked  anomaly  list. 
Category  -1,  0,  1,  2  anomalies  were  placed  above  the  dig  threshold  in  descending  order,  while 
Category  3  anomalies  were  placed  below  the  dig  threshold.  The  first  items  on  each  ranked 
anomaly  list  were  those  targets  for  which  ground  truth  labels  were  requested  (training  data). 
Following  this,  anomalies  for  which  reliable  parameters  could  not  be  extracted  and  therefore  had 
to  be  dug  were  listed.  Next  were  the  items  that  Arcadis  was  the  most  confident  are  “highly  likely” 
to  be  TOI.  The  items  were  ranked  in  descending  confidence  that  the  item  is  TOI.  Any  items  that 
the  analyst  could  analyze,  but  were  not  able  to  classify  (i.e..  Category  2  anomalies)  were  placed 
next  on  the  anomaly  list.  Finally,  all  Category  3  items  that  the  analyst  was  confident  were  not  TOI 
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were  ranked  in  descending  order  of  their  confidence  metric.  The  Category  3  anomalies  were 
ordered  in  descending  order  based  on  their  confidence  metric. 

Table  6-1:  Example  Ranked  Anomaly  List 


Target 

ID 

Category 

Dig 

Decision 
(l=Dig; 
0=Do 
Not  Dig) 

Type 

(mm) 

Confidence 

Metric 

Comment 

CR-24 

-1 

1 

127 

1 

Training  Data 

Training  Data 

CR-339 

-1 

1 

127 

0.65 

Training  Data 

CR-17 

0 

1 

127 

9999 

Can't  Analyze 

CR-862 

0 

1 

127 

9999 

Can't  Analyze 

CR-167 

2 

1 

60 

0.8 

Cannot  Decide 

CR-710 

3 

0 

0 

1 

High  Confidence  Match  to 
Clutter  Items 

CR-1006 

3 

0 

0 

0.8 

High  Confidence  Match  to 
Clutter  Items 

CR-631 

3 

0 

0 

0.79 

Low  Confidence  Match  to 
TOI  and  Clutter  Items 

CR-978 

3 

0 

0 

0 

Low  Confidence  Match  to 
TOI  and  Clutter  Items 

Note: 

mm  -  millimeter 


In  addition,  Arcadis  provided  an  assignment  to  the  ‘Type’  column,  indicating  the  specific  type  of 
munition  caliber  (i.e.,  75  mm).  The  final,  ranked  anomaly  lists  are  provided  in  Appendix  B. 

6.5.2  Feedback  Based  on  QC  Seed  Performance 

The  initial  ranked  anomaly  lists  were  submitted  to  the  ESTCP  program  office  to  confirm  that  all 
QC  seeds  had  been  classified  correctly.  QC  seed  items  were  not  marked  to  be  dug  on  the  initial 
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dig  list;  therefore,  the  ESTCP  Program  Office  provided  the  location  and  identity  of  the 
misclassified  QC  seeds  as  additional  on-site  training  data.  Arcadis  used  this  additional  information 
to  modify  our  classification  procedures  to  ensure  that  the  revised  methods  correctly  classified  the 
seeds  missed  originally.  A  second  stage  list  was  then  submitted  accompanied  by  a  Failure  Analysis 
memo  outlining  the  causes  of  the  misclassification,  the  revisions  made  in  classification  procedures, 
and  a  demonstration  that  the  revised  procedures  successfully  classified  the  missed  seeds. 
Appendix  B  contains  the  QC  Seed  Failure  Memos  for  the  TEMTADS  dig  lists  for  both  Analyst  1 
and  2.  Upon  acceptance  of  the  memos  by  the  ESTCP  Program  Office,  the  analysts  submitted  Final 
Ranked  Dig  Lists  (see  Appendix  B)  to  the  Program  Office  and  IDA  scored  the  second  stage  list. 


6.6  DATA  PRODUCTS 

6.6.1  Ranked  Anomaly  Lists  and  Results 

As  discussed  above,  each  Arcadis  Analyst  submitted  a  ranked  anomaly  list  for  the  cued 
TEMTADS  data  (see  Appendix  B).  Upon  submittal  of  the  final  ranked  dig  lists,  IDA  constructed 
Receiver  Operating  Characteristic  (ROC)  curves  for  each  anomaly  list.  Figure  6-1  and  Figure  6- 
2  show  the  ROC  curves  for  Analyst  1  and  Analyst  2,  respectively. 

Figure  6-1:  Analyst  1  ROC  Curve 


FtBliss  Arcadisl  LibMatch  None  2x2  Custom  s2  v2 


Difficult  TOIs: 

#CR-930  (25mm  legacy) 
#CR-692  (37mm  legacy) 
#CR-1089  (37mm  legacy) 
#CR-198  (37mm  legacy) 
#CR-886  (37mm  legacy) 
#CR-1 148  (37mm  legacy) 
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Figure  6-2:  Analyst  2  ROC  Curve 
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Difficult  TOIs: 

#CR-930  (25mm  legacy) 
#CR-662  (37mm  legacy) 
#CR-1089  (37mm  legacy) 
#CR-692  (37mm  legacy) 
#CR-886  (37mm  legacy) 
#CR-1 148  (37mm  legacy) 


Number  of  Non-TOIs  Incorrectly  Classified 


The  key  regions  to  interpret  the  ROC  curves  are: 

•  The  black  dashed  line  corresponds  to  the  targets  that  were  dug  for  training  data. 

•  The  solid  black  line  corresponds  to  the  targets  that  were  categorized  as  “can’t  extract 
reliable  parameters”  that  were  treated  as  potential  TOI  because  no  meaningful 
classification  could  be  done.  At  least  one  TOI  was  identified  in  this  part  of  the  curve  for 
each  of  the  three  ROC  curves. 

•  Targets  in  red,  yellow,  green,  correspond  to  Category  1,  2,  and  3  targets,  respectively. 

•  The  blue  dot  corresponds  to  the  dividing  point  between  TOI  and  not-TOI. 

•  The  orange  dot  corresponds  to  the  point  at  which  ARCADIS  detected  all  TOI. 

As  discussed  in  Section  3.0,  the  primary  performance  metric  was  the  point  at  which  the  ROC  curve 
reaches  100%  identification  of  TOI.  The  number  of  clutter  items  correctly  identified  at  this  point 
is  a  measure  of  the  savings  possible  for  each  method.  As  seen  in  Figures  6-1  and  6-2,  both  Arcadis 
analysts  had  6  difficult  to  detect  targets  that  were  incorrectly  classified  as  non-TOI.  Section  7.0 
contains  an  assessment  as  to  why  these  were  incorrectly  classified  as  non-TOI. 

6.6.2  Intrusive  Investigation  Results 

See  the  ESTCP  Program  Office  report  for  intrusive  investigation  results. 
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7.0  PERFORMANCE  ASSESSMENT 


Performance  objectives  were  established  in  the  demonstration  plan  to  evaluate  the  quality  of  data 
collected  as  part  of  this  demonstration.  These  performance  objectives  were  first  discussed  in 
Section  3.0  of  this  report.  This  section  documents  the  results  and  evaluates  the  data  quality  and 
whether  the  performance  metrics  were  met.  Table  7-1  shows  the  performance  results  for  each  of 
the  performance  objectives  for  both  analysts,  while  the  following  sub-sections  describe  the 
performance  relative  to  the  MQOs.  Where  the  results  of  the  two  analysts  vary,  additional  sub¬ 
sections  have  been  included  to  document  the  results  of  both  analysts. 

Table  7-1:  Cued  TEMTADS  Data  Collection  and  Analysis  Objectives 


Performance 

Objective 

Metric 

Data 

Required 

Success 

Criteria 

Analyst  1 
Results 

Analyst  2 
Results 

Confirm  all 
background 
measurements 
are  valid 

Sensor 

response 

Background 

measurements 

All  decay 
amplitudes  lower 
than  project 
threshold  and 
qualitatively  agree 
with  initial 

measurement 

MQO 
Achieved, 
61%  of 
backgrounds 
rejected 

MQO 
Achieved, 
64%  of 
backgrounds 
rejected 

Confirm  all 

measurements 
have  an 
applicable 
background 

Background 

measurements 

Cued  survey 
data 

Time  Separation 
between 
background 
measurement  and 
anomaly 

measurement  <  2- 
hrs 

MQO  not 
achieved, 
84.3%  <  2- 
hrs  after 
removing 
bad 

backgrounds 

MQO  not 
achieved, 
80.6%  <  2- 
hrs  after 
removing 
bad 

backgrounds 

Valid  Position 
Data  (1): 
Global 
Positioning 
System  (GPS) 

GPS  status 
flag 

Cued  survey 
data 

Confirm  GPS 
status  flag 
indicates  RTK  fix 

MQO 

achieved, 

100% 

MQO 

achieved, 

100% 

Valid  Position 
Data  (2):  IMU 

IMU 

Orientation 

data 

Cued  survey 
data 

Orientation  data 
valid  data  input 
string  checksum 
passes 

MQO 

achieved, 

100% 

MQO 

achieved, 

100% 

Confirm 
inversion 
model 
supports 
classification 
(1  of  3) 

Fit  Coherence 

Cued  survey 
data 

Derived  model 
response  must  fit 
the  observed  data 
with  a  fit 
coherence  >0.8 

MQO  not 
achieved 
98%  of 
targets 

MQO  not 
achieved 
98%  of 
targets 

Confirm 

inversion 

Fit  location 

Cued  survey 
data 

Fit  location 
estimate  of  item  < 

MQO 

achieved 

MQO 

Achieved 
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Performance 

Objective 

Metric 

Data 

Required 

Success 

Criteria 

Analyst  1 
Results 

Analyst  2 
Results 

model 
supports 
classification 
(2  of  3) 

0.4m  from  center 
of  sensor 

Confirm 
inversion 
model 
supports 
classification 
(3  of  3) 

Fit  Location 

Cued  survey 
data 

100%  of  blind 
seed  item 
positions  <  0.25  m 
from  predicted 
position  (x,  y,  z) 

MQO  not 
achieved, 
71.4%  < 
0.25-m 

MQO  not 
achieved, 
75%  <  0.25- 

m 

Confirm 
derived 
features  match 
ground  truth  (1 
of  2) 

Fit  Location 

Prioritized 
classification 
dig  list  and 
dig  results 

100%  of 
recovered 
(excluding  can’t 
analyze  category) 
item  positions  < 
0.25  m  from 
predicted  position 
(x,  y) 

MQO  not 
achieved, 
80%  <  0.25- 

m 

MQO  not 
achieved, 
66%  <0.25-m 

Confirm 
derived 
features  match 
ground  truth  (2 
of  2) 

Classification 

results 

Prioritized 
classification 
dig  list  and 
dig  results 

100%  of 
recovered  object 
size  estimates 
(excluding  can’t 
analyze  category) 
qualitatively 
match  predicted 
size 

MQO  not 
achieved 

MQO  not 
achieved 

Validation  of 
TOI/non-TOI 
thresholds 

Classification 

results 

Prioritized 
classification 
dig  list  and 
dig  results 

100%  of  predicted 
non-TOI 
intrusively 
investigated  are 
non-TOI 

MQO  not 
achieved, 
83%  of  TOI 
correctly 
classified 

MQO  not 
achieved, 
83%  of  TOI 
correctly 
classified 

7.1  OBJECTIVE:  CONFIRM  ALL  BACKGROUND  MEASUREMENTS  ARE  VALID 

Throughout  the  cued  data  collection  process,  URS  collected  ongoing  background  measurements 
approximately  every  2  hours  to  minimize  the  time-variable  changes  in  the  background  response. 
To  maintain  production,  the  background  location  selected  for  a  given  background  measurement 
were  based  on  proximity  to  the  location  of  cued  data  acquisition.  This  resulted  in  a  total  180 
background  readings  collected  at  15  different  background  locations  (3  located  within  the  IVS  area, 
6  located  within  grid  areas,  and  6  located  outside  of  grid  areas)  over  the  course  of  cued  data 
acquisition.  The  reliability  of  advanced  EMI  sensor  cued  data  collection  and  processing  is 
foundationally  dependent  on  acquiring  background  measurements  that  are  free  of  metallic  objects 
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that  could  adversely  affect  the  background  subtraction  process.  However,  over  the  course  of  cued 
data  collection,  the  data  collection  contractor  did  not  pay  enough  attention  to  this  detail,  resulting 
in  background  locations  where  metal  objects  were  located  less  than  5  feet  away  from  some  and 
less  than  1  foot  in  others.  Figure  7-1  shows  an  example  of  the  variation  in  background  response 
at  background  location  2  for  transmit  coil  4.  Ideally,  there  would  be  very  little  variation, 
particularly  in  the  early  time  gates  (e.g.,  before  1  millisecond[ms]).  However,  it  is  apparent  from 
the  figure  that  there  are  large  responses  (e.g.,  10  milliVolt  [mV]/amp  in  the  4Z_3x  response)  that 
indicate  there  are  metallic  objects  near  the  sensor.  These  metallic  objects  should  have  been 
removed  from  the  area  prior  to  use  this  particular  background  location. 

Figure  7-1:  Example  of  Background  Variability  at  Background  Location  2 
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Because  of  the  metallic  objects  near  background  locations,  7  of  the  15  total  background  locations 
were  deemed  to  be  compromised  after  an  initial  visual  inspection  of  the  decay  amplitudes.  The 
remaining  background  readings  from  the  8  acceptable  background  locations  were  further 
inspected,  and  all  quantitative  and  qualitative  outliers  were  removed  from  the  list  of  available 
background  readings.  Analyst  1  determined  that  only  69  of  the  160  background  measurements 
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(38.33%)  were  acceptable  to  level  the  cued  data.  Analyst  2  used  the  same  evaluation  process  as 
Analyst  1,  but  determined  that  only  64  of  the  180  background  measurements  (35.6%)  were 
acceptable  to  level  the  cued  data. 

7.2  OBJECTIVE:  CONFIRM  ALL  MEASUREMENTS  HAVE  AN  APPLICABLE 
BACKGROUND 

7.2.1  Analyst  1 

The  Final  QAPP  and  SOP  MEC-AC-12  specify  that  the  time  separation  between  background 
measurements  should  be  less  than  2  hours  to  minimize  the  time-variable  changes  in  the  background 
response.  However,  as  stated  in  Section  7.1,  after  the  background  QC  process  was  completed, 
Analyst  1  determined  that  only  69  out  of  the  total  180  background  measurements  (38.33%)  were 
acceptable  for  background  subtraction.  Figure  7-2  shows  a  summary  of  the  time  between 
background  readings  that  Analyst  1  determined  were  acceptable  and  for  all  the  cued  data.  Only 
84.3%  of  the  1879  cued  measurements  (including  re-shots)  had  an  acceptable  background  reading 
collected  within  the  2-hour  window.  This  resulted  in  the  use  of  background  readings  far  outside  of 
the  typical  2-hour  window  (up  to  5  hours)  to  complete  the  cued  data  analysis. 

Figure  7-2:  Analyst  1  Background  Reading  Time  Window  Distribution 


28 


7.2.2  Analyst  2 


The  Final  QAPP  and  SOP  MEC-AC-12  specify  that  the  time  separation  between  background 
measurements  should  be  less  than  2  hours  to  minimize  the  time-variable  changes  in  the  background 
response.  However,  as  stated  in  Section  7.1.2,  after  background  QC  process  was  completed,  only 
64  out  of  the  total  180  background  measurements  (36%)  were  deemed  acceptable  for  background 
subtraction.  Figure  7-3  shows  a  summary  of  the  time  between  background  readings  that  Analyst 
2  determined  were  acceptable  and  for  all  the  cued  data.  Only  80.6%  of  the  1,879  cued 
measurements  (including  re-shots)  had  an  acceptable  background  reading  collected  within  the  2- 
hour  window.  This  resulted  in  the  use  of  background  readings  far  outside  of  the  typical  2-hour 
window  (up  to  6  hours)  to  complete  the  cued  data  analysis. 


Figure  7-3:  Analyst  2  Background  Reading  Time  Window  Distribution 


7.3  OBJECTIVE:  VALID  POSITION  DATA  (1)  -  GPS 

The  RTK  fix  quality  flag  was  used  to  evaluate  whether  each  cued  measurement  had  a  valid  RTK 
GPS  location  recorded  with  it.  All  cued  data  had  an  RTK  fix  quality  of  4,  indicating  that  valid 
positioning  was  recorded  for  each  measurement.  All  data  passes  this  perfonnance  metric  for  both 
analysts. 
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7.4  OBJECTIVE:  VALID  POSITION  DATA  (2)  -  IMU 


During  data  acquisition,  the  field  software  was  monitored  to  ensure  that  all  data  streams  ( e.g .,  GPS, 
IMU)  were  valid  and  being  recorded.  The  objective  was  to  record  a  valid  data  stream  from  the 
IMU  for  each  cued  measurement.  The  IMU  data  stream  recorded  in  each  cued  measurement  were 
evaluated  by  checking  the  raw  data  import  for  a  valid  value  in  each  of  the  UXA_pitch,  UXA_yaw, 
and  UXA  roll  database  channels.  This  performance  metric  was  met  for  both  analysts,  as  valid 
pitch,  roll,  and  yaw  data  streams  were  recorded  for  each  cued  measurement. 

7.5  OBJECTIVE:  CONFIRM  INVERSION  MODEL  SUPPORTS  CLASSIFICATION 
(1  OF  3)  -  FIT  COHERENCE 

The  reliability  of  the  classification  process  depended  on  collecting  high  quality  data  to  support  the 
modeling  process.  This  objective  evaluated  how  well  the  modeled  results  of  the  inversion  process 
correlated  to  the  observed  data.  A  fit  coherence  metric  (UXA  Fit  Coh)  is  calculated  for  each 
model  during  data  inversion,  and  is  used  as  a  basis  for  determining  how  well  the  model  matches 
the  observed  data. 

Each  cued  measurement  was  assessed  by  evaluating  the  coherencies  derived  for  single  source  and 
multi-source  fits.  The  highest  fit  coherency  value  from  either  solver  was  used  as  the  basis  for 
evaluation.  Modeled  results  with  a  neither  a  single  source  or  multi-source  fit  coherence  greater 
than  0.8  were  placed  in  the  ‘cannot  analyze’  category. 

This  objective  requires  that  the  model  derived  during  the  inversion  process  match  the  observed 
data  to  within  a  correlation  of  0.8.  This  metric,  provided  by  UX  Analyze  as  the  ‘fit  coherence’ 
(UXA  Fit  Coh)  is  an  output  of  the  dipole  analysis  inversion  routine. 

For  both  Arcadis  analysts,  98%  of  the  inversion  results  resulted  in  fit  coherences  greater  than  0.8. 
The  fit  coherences  were  likely  decreased  due  to  the  large  time  distance  between  cued 
measurements  and  acceptable  background  data,  as  well  as  to  the  overall  quality  of  the  background 
measurements. 

7.6  OBJECTIVE:  CONFIRM  INVERSION  MODEL  SUPPORTS  CLASSIFICATION 
(2  OF  3)  -  FIT  LOCATION 

The  reliability  of  the  classification  results  also  depended  on  collecting  data  where  the  sensor  is 
above  the  subsurface  metallic  object.  The  metric  for  this  objective  was  met  if  the  center  of  the 
instrument  was  positioned  within  40-cm  of  the  actual  anomaly  location  for  100%  of  the  cued 
anomalies.  Single  and  multi-source  fit  locations  were  evaluated  for  each  cued  measurement 
location  to  assess  this  metric.  Based  on  the  fit  results,  100%  of  the  cued  measurements  evaluated 
during  this  demonstration  had  a  fit  location  within  40-cm  of  the  recorded  location  of  the 
TEMTADS  2x2  array  center  for  both  analysts. 
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7.7 


OBJECTIVE:  CONFIRM  INVERSION  MODEL  SUPPORTS  CLASSIFICATION 
(3  OF  3)  -  FIT  LOCATION 


7.7.1  Analyst  1 

The  reliability  of  the  classification  results  also  depended  on  reliably  being  able  to  classify  targets 
at  their  respective  location.  The  metric  for  this  objective  was  met  if  100%  of  the  modelled  seed 
locations  were  less  than  or  equal  to  0.25-m  from  their  known  position  (x,  y,  z). 

For  Analyst  1,  20  of  the  28  total  seed  items  (71.4%)  met  this  criterion.  For  Analyst  2,  21  of  the  28 
total  seed  items  (75%)  met  this  criterion.  Thus,  both  analysts  did  not  meet  this  MQO.  This  failure 
was  likely  due  to  a  combination  of  inconsistent  background  readings  (taken  greater  than  2  hours 
apart)  and  the  abundance  of  signal  interference  from  the  presence  of  immediately  adjacent  shallow 
clutter/munitions  debris  (frag  and  fuse  components).  Figure  7-4  and  Figure  7-5  present  a 
summary  of  offsets  between  the  fit  locations  of  the  inverted  data  and  the  seed  item  locations  for 
analysts  1  and  2,  respectively. 

Figure  7-4:  Analyst  1  Predicted  Seed  Location  Distribution 
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Figure  7-5:  Analyst  2  Predicted  Seed  Location  Distribution 


7.8  OBJECTIVE:  CONFIRM  DERIVED  FEATURES  MATCH  GROUND  TRUTH  (1 
OF  2) 

The  reliability  of  the  classification  results  depended  on  reliably  being  able  to  classify  targets  at 
their  respective  location.  The  performance  objective  was  met  if  100%  of  the  predicted  target 
locations  of  all  items  recovered  from  target  locations  in  the  training  set  (-1),  likely  TOI  (1),  and 
can’t  decide  (2)  class  of  the  final  dig  list  were  less  than  or  equal  to  0.25  m  from  known  position 
(x,  y,  z).  Figure  7-6  and  Figure  7-7  show  summaries  of  the  offsets  between  the  fit  locations  of  the 
inverted  data  and  the  locations  of  the  recovered  items  within  dig  classes  -1,1,  and  2  for  analysts  1 
and  2,  respectively.  They  respectively  show  that  80%  and  66%  of  the  modeled  target  locations  for 
Analysts  1  and  2  met  this  criterion,  thus  neither  analyst  achieved  this  MQO.  This  failure  was  likely 
largely  due  to  the  abundance  of  shallow  clutter  and  munitions  debris  recovered  within  the  dig 
proximity  of  the  primary  target  of  concern  and  possibly  due  to  the  extended  period  between 
background  readings  after  removing  bad  background  readings. 
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Figure  7-6:  Analyst  1  Target  Location  Offset  Distribution 


Figure  7-7:  Analyst  2  Target  Location  Offset  Distribution 
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7.9  OBJECTIVE:  CONFIRM  DERIVED  FEATURES  MATCH  GROUND  TRUTH  (2 
OF  2) 

The  reliability  of  the  classification  results  depended  on  reliably  being  able  to  classify  the  size  of 
targets.  The  objective  was  met  if  100%  of  recovered  object  size  estimates  (excluding  can’t  analyze 
category)  qualitatively  match  predicted  size.  The  perfonnance  metric  was  met  if  100%  of  the 
recovered  object  size  estimates  qualitatively  matched  the  predicted  sizes.  To  assess  this,  the 
predicted  sizes  of  the  targets  in  Class  1  (likely  TOI)  and  2  (can’t  decide)  on  the  final  submitted  dig 
list  were  compared  to  the  dig  results  in  the  training  data  for  the  site  provided  by  ESTCP.  The  size 
bands  approximated  on  the  dig  list  were  based  on  object  diameter  and  ranged  from  0  to  50-mm 
(size  band  1),  between  50  mm  and  100  mm  (size  band  2),  and  greater  than  100  mm  (size  band  3). 

7.9.1  Analyst  1 

Of  the  465  targets  Analyst  1  classified  as  class  1  and  2  targets  on  the  final  dig  list,  298  targets  were 
interpreted  as  size  band  1,  164  targets  were  interpreted  as  size  band  2,  and  3  targets  were 
interpreted  as  size  band  3.  From  a  qualitative  perspective,  all  the  items  recovered  from  target 
locations  predicted  to  contain  objects  in  size  band  1  were  consistent  with  smaller  diameter  objects: 
munition  fragments,  small  anns,  fuse  components,  37-mm  rounds,  and  small  ISO  seeds.  In 
contrast,  the  items  recovered  from  target  locations  predicted  to  contain  objects  with  diameters  in 
size  band  2  constraints  were  not  consistent  with  medium  diameter  objects,  also  consisting  of 
munition  fragments,  small  arms,  fuse  components,  and  several  small  ISO  seeds.  However,  the  3 
targets  selected  to  be  in  size  band  3  did  consist  of  large  munitions  debris.  Analyst  1  didn’t  meet 
this  MQO  based  on  the  relatively  small  items  found  in  size  band  2. 

7.9.2  Analyst  2 

Of  the  483  targets  Analyst  2  classified  as  class  1  and  2  targets  on  the  final  dig  list,  327  were 
interpreted  as  size  band  1,  155  were  interpreted  as  size  band  2,  and  1  was  interpreted  as  size  band 
3.  From  a  qualitative  perspective,  all  the  items  recovered  from  target  locations  predicted  to  contain 
objects  in  size  band  1  were  consistent  with  smaller  diameter  objects:  munition  fragments,  small 
anns,  fuse  components,  37-mm  rounds,  and  small  ISO  seeds.  In  contrast,  the  items  recovered  from 
target  locations  predicted  to  contain  objects  with  diameters  in  size  band  2  constraints  were  not 
consistent  with  medium  diameter  objects,  also  consisting  of  munition  fragments,  small  arms,  fuse 
components,  and  several  small  ISO  seeds;  as  well  as  0.50  caliber  projectile.  However,  the  only 
target  selected  to  be  in  size  band  3  did  consist  of  a  105mm  projectile.  Analyst  2  didn’t  meet  this 
MQO  based  on  the  relatively  small  items  found  in  size  band  2. 

7.10  OBJECTIVE:  VALIDATION  OF  TOI/NON-TOI  THRESHOLDS 

The  data  reliability  depends  on  the  classification  approach  to  correctly  identify  TOI  and  non-TOI. 
The  objective  was  met  if  100%  of  predicted  non-TOI  intrusively  investigated  were  non-TOI. 

Analysts  1  and  2  prepared  ranked  anomaly  lists  for  the  cued  TEMTADS  dataset  and  IDA  personnel 
used  their  scoring  algorithms  to  assess  the  results.  The  objective  was  met  if  all  the  TOI  were 
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correctly  labeled  as  TOI  on  the  ranked  anomaly  list.  Figure  6-1  and  Figure  6-2  show  the  ROC 
curves  for  Analysts  1  and  2,  while  Table  7-2  presents  a  summary  of  the  results.  As  seen  in  the 
ROC  curves,  Analysts  1  and  2  both  incorrectly  classified  6  TOI  as  non-TOI.  The  following  sub¬ 
sections  present  assessments  for  why  each  of  the  TOI  were  misclassified. 


Table  7-2:  Prioritized  Dig  List  Statistics  for  Analysts  1  and  2 


Analyst 

TOI 

Identified 

Training 

Targets 

Cannot 

Analyze 

List  Length 

Totals 

Qty- 

% 

Qty. 

% 

Qty. 

% 

Qty. 

% 

Total 

TOI 

Total 

Targets 

Analyst  1 

34 

82.9% 

81 

5.4% 

105 

7.0% 

651 

43.7% 

41 

1491 

Analyst  2 

34 

82.9% 

42 

2.8% 

100 

6.7% 

629 

42.2% 

7.10.1  Analyst  1 

Figure  6-1  shows  the  ROC  curve  for  Analyst  1 .  As  seen  on  this  figure,  Analyst  1  incorrectly 
classified  six  TOI  as  non-TOI,  which  are  listed  below: 

•  CR-930:  25mm  projectile 

•  CR-692:  37-inm  projectile 

•  CR-1089:  37-mm  projectile 

•  CR-198:  37-mm  projectile 

•  CR-886:  37-mm  projectile 

•  CR-1 148:  37-mm  projectile. 

Initial  Category  1  and  2  target  selection  thresholds  were  detennined  based  on  standard  advanced 
classification  processing  recommendations  made  by  Geosoft  (0.925  to  0.825).  However,  given 
the  elevated  levels  of  background  geologic  noise,  as  well  as  the  reduced  number  of  acceptable 
background  measurements  and  the  failure  to  correctly  classify  a  QC  seed  as  TOI,  the  following 
decision  statistics  were  used  to  classify  targets  as  Category  1  through  3  targets  in  the  final  dig  list: 

•  Category  1 :  Anomalies  that  had  a  combined  metric  match  to  TOI  greater  than  0.925 .  These 
anomalies  were  believed  to  have  a  high  likelihood  of  being  TOI. 

•  Category  2:  These  anomalies  were  anomalies  for  which  Analyst  1  was  unsure  whether  the 
anomaly  was  TOI  or  non-TOI  and  were  classified  as  follows: 

o  Targets  that  met  all  criteria  in  Table  1  in  the  Analyst  1  Anomaly  Decision  Memo 
(See  Appendix  B)  with  decision  statistics  between  0.825  and  0.925,  and 
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o  Non-weak  targets  that  had  decision  statistics  above  0.75. 

•  Category  3:  Anomalies  that  had  a  combined  metric  match  to  TOI  that  didn’t  meet  any  of 
the  above  criteria.  These  anomalies  had  a  high  likelihood  of  being  clutter  items  and/or  a 
low  likelihood  of  being  TOI.  These  anomalies  were  placed  below  the  dig  threshold. 

Can’t  Analyze  anomalies  were  determined  based  on  one  or  more  of  the  following  criteria: 

•  There  were  no  polarizabilities  because  the  inversion  did  not  finish; 

•  The  fit  depth  was  unreasonable  ( e.g .,  greater  than  2-m  below  ground  surface  [bgs]) 

•  There  were  negative  polarizabilities; 

•  There  was  a  poor  fit  coherence  (e.g.,  less  than  0.8);  and 

•  Reliable  polarizabilities  couldn’t  be  extracted  (e.g.,  array  center  to  fit  location  greater 
than  0.4-m). 

In  Analyst  1  ’s  final  dig  list,  the  six  targets  in  question  had  a  decision  statistic  range  of  0.53  through 
0.76,  and  as  such,  were  classified  as  Category  3  targets  and  not  recommended  to  be  dug.  Table 
7-3  shows  the  initial  decision  statistic  and  the  best  3-component  match  for  each  of  the  missed  TOI. 
The  best  fits  included  3  rifle  grenades,  2  37-mm  projectiles,  and  1  40-mm  projectile.  A  visual 
inspection  of  the  six  recovered  items  shows  that  except  for  the  item  encountered  at  target  CR-0930 
(a  25-inm  projectile),  all  the  remaining  targets  consisted  of  large  remnants  of  37-mm  projectiles, 
and  as  such,  were  classified  as  37-mm  munitions  debris  (MD)  in  the  field.  However,  it  was  later 
detennined  by  the  site  team  that  these  items  were  close  enough  to  a  complete  37-mm  projectile  to 
be  considered  TOI  for  the  purposes  of  AGC. 

Table  7-3:  Analyst  1  Decision  Statistic  Comparison 


Target 

ID 

Initial  AGC  Results 

Revised  AGC 

Decision 

Statistic 

Best  Match 

Decision 

Statistic 

Best  Match 

CR-930 

0.6119 

37-mm 

0.6925 

20-mm  projectile 

CR-692 

0.6568 

Steel  rifle  grenade 

0.8861 

20-mm  projectile 

CR-1089 

0.5275 

37-mm 

0.9139 

20-mm  projectile 

CR-198 

0.6775 

Steel  rifle  grenade 

0.8899 

20-mm  projectile 

CR-886 

0.7508 

Steel  rifle  grenade 

0.9356 

20-mm  projectile 

CR-1148 

0.7618 

40-mm 

0.8271 

20-mm  projectile 

While  20-mm  projectiles  were  initially  included  in  the  field  verification  and  validate  library 
versions  of  the  TOI  library,  they  were  removed  from  the  TOI  library  for  the  final  classification  to 
reduce  the  number  of  false  matches  to  clutter  objects  after  confirming  with  the  ESTCP  Program 
Office  that  20-mm  projectiles  were  not  TOI  at  CR.  After  failing  to  identify  the  6  missed  TOI,  the 
six  targets  in  question  were  re-processed  with  20-mm  projectiles  included  in  the  TOI  library. 
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Table  7-3  shows  the  revised  AGC  results  for  the  6  misclassified  targets,  which  were  assigned  a 
decision  statistic  between  0.69  through  0.94,  which  all  fall  above  the  stop  dig  threshold  except  for 
target  CR-930.  Analyst  1  would  have  needed  to  include  20-mm  projectiles  in  the  TOI  library  and 
reduce  the  stop-dig  threshold  to  0.69  to  correctly  classify  all  the  TOI.  Reducing  the  stop-dig 
threshold  to  0.69  would  have  significantly  increased  the  number  of  non-TOI  targets  that  required 
intrusive  investigation  to  ensure  all  TOI  were  recovered.  See  Figure  7-8  through  Figure  7-13  for 
the  revised  library  match  for  each  of  the  missed  TOI. 

7.10.2  Analyst  2 

Figure  6-2  shows  the  ROC  curve  for  Analyst  2.  As  seen  on  this  figure,  Analyst  2  incorrectly 
classified  six  TOI,  which  are  listed  below: 

•  CR-930:  25mm  projectile 

•  CR-662:  37-min  projectile 

•  CR- 1089:  3 7-mm  projectile 

•  CR-692:  3 7-mm  projectile 

•  CR-886:  3 7-mm  projectile 

•  CR-1 148:  3 7-mm  projectile. 

Initial  Category  1  and  2  target  selection  thresholds  were  detennined  based  on  standard  advanced 
classification  processing  recommendations  made  by  Geosoft  (0.925  to  0.75).  However,  given  the 
elevated  levels  of  background  geologic  noise,  as  well  as  the  reduced  number  of  acceptable 
background  measurements,  the  following  decision  statistics  were  used  to  classify  targets  as 
Category  1  through  3  targets  in  the  final  dig  list: 

•  Category  1  targets  are  those  with  decision  statistics  greater  than  0.925. 

•  Category  2  targets  are  those  with  decision  statistics  between  0.925  and  0.7 1 . 

•  Category  3  targets  are  those  with  decision  statistics  below  0.71. 

Can’t  Analyze  anomalies  were  determined  based  on  one  or  more  of  the  following  criteria: 

•  There  were  no  polarizabilities  because  the  inversion  did  not  finish; 

•  The  fit  depth  was  unreasonable  (e.g.,  greater  than  2-m  bgs) 

•  There  were  negative  polarizabilities; 

•  There  was  a  poor  fit  coherence  (e.g.,  less  than  0.8);  and 

•  Reliable  polarizabilities  couldn’t  be  extracted  (e.g.,  array  center  to  fit  location  greater 
than  0.4-m). 

In  Analyst  2’s  final  dig  list,  the  six  targets  in  question  had  a  decision  statistic  range  of  0.42  through 
0.69,  and  as  such,  were  classified  as  Category  3  targets  and  not  recommended  to  be  dug,  although 
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Figure  7-8:  Analyst  1  Revised  Library  Match  for  Target  CR-930 
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Figure  7-9:  Analyst  1  Revised  Library  Match  for  Target  CR-692 
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Figure  7-10:  Analyst  1  Revised  Library  Match  for  Target  CR-1089 
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Figure  7-11:  Analyst  1  Revised  Library  Match  for  Target  CR-198 
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Figure  7-12:  Analyst  1  Revised  Library  Match  for  Target  CR-886 
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Figure  7-13:  Analyst  1  Revised  Library  Match  for  Target  CR-1148 
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it  should  be  noted  four  of  the  six  targets  matched  best  to  a  37-nnn  projectile  (the  remaining  two 
matched  to  a  steel  rifle  grenade).  A  visual  inspection  of  the  six  recovered  items  shows  that  except 
for  the  item  encountered  at  target  CR-0930  (a  25-mm  projectile),  all  the  remaining  targets 
consisted  of  large  remnants  of  37-mm  projectiles,  and  as  such,  were  classified  as  37-mm  MD  in 
the  field.  However,  it  was  later  detennined  by  the  site  team  that  these  items  were  close  enough  to 
a  complete  37-mm  projectiles  to  be  considered  TOI.  Additionally,  while  20-inm  projectiles  were 
initially  included  in  the  field  check  and  validate  library  versions  of  the  TOI  library,  after 
confirming  with  ESTOP  that  no  20-nnn  projectiles  were  used  on  the  site,  they  were  not  included 
in  the  final  version  of  the  TOI  library  used  to  classify  and  rank  the  targets  prior  to  submitting  the 
final  dig  list,  to  reduce  false  matches  to  fragmented  clutter.  When  the  six  targets  in  question  were 
re-processed  with  20-mm  projectile  included  in  the  final  TOI  library,  these  targets  were  assigned 
a  revised  UXA  decision  statistic  range  of  0.72  through  0.94,  which  now  fall  within  the  boundary 
of  Category  1  and  2  targets  (decision  statistic  greater  than  0.71),  and  were  all  classified  as  20-mm 
projectiles  (see  the  polarization  plots  below  presented  as  Figure  7-14  through  Figure  7-19. 

In  summary,  Analyst  2  incorrectly  classified  the  6  TOI  because  they  all  consisted  of  incomplete 
rounds,  and  that  20-mm  projectiles  were  not  included  in  the  final  TOI  library.  Although  none  of 
the  items  recovered  were  20-mm  projectiles,  the  20-mm  projectile  was  their  best  match. 

Table  7-4:  Analyst  2  Decision  Statistic  Comparison 


Target  ID 

Initial  AGO  Results 

Revised  AGC 

Decision 

Statistic 

Best  Match 

Decision 

Statistic 

Best  Match 

CR-930 

0.6463 

37-mm  projectile 

0.7222 

20-mm  projectile 

CR-662 

0.6528 

Steel  Rifle  Grenade 

0.8705 

20-mm  projectile 

OR- 1089 

0.4176 

37  mm  projectile 

0.9119 

20-mm  projectile 

CR-692 

0.5568 

37-mm  projectile 

0.8365 

20-mm  projectile 

CR-886 

0.6899 

Steel  Rifle  Grenade 

0.9360 

20-mm  projectile 

OR- 1148 

0.7113 

37-mm  projectile 

0.8609 

20-mm  projectile 
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Figure  7-14:  Analyst  2  Revised  Library  Match  for  CR-930 
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Figure  7-15:  Analyst  2  Revised  Library  Match  for  CR-662 
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Figure  7-16:  Analyst  2  Revised  Library  Match  for  CR-1089 


1089  001  02 


20mm_BP1 1 1 103_TP4s 
UXA_LmStat_1 1 1  toi  -  0.89264 


20mm_BP1 1 1 103_TP36s 
UXA_LmStat_1 1 1  toi  -  0.781 5 


10'2  10’’  10°  10'  id2 


Tn*  (m») 


T*n*  (m») 


20mm_BP1 1 1 103_TP4s 
UXA_LmStat_1 1 0toi  -  0.85166 


10'2  10''  10°  io’  102 


20mm_BP1 1 1 103_TP36s 
UXA_LmStat_1  lOtoi  -  0.78618 


10'2  10'1  10°  io'  102 


Tm*  (ms) 


Tme(ro) 


47 


Figure  7-17:  Analyst  2  Revised  Library  Match  for  CR-692 
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Figure  7-18:  Analyst  2  Revised  Library  Match  for  CR-886 
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Figure  7-19:  Analyst  2  Revised  Library  Match  for  CR-1148 


1148  001  01 


20mm_BP1 1 1 103_TP2s 
UXA_LmStat_1 1 1  toi  -  0.75702 


20mm_BP1 1 1 103_TP4s 
UXA_LmStat_1 1 1  toi  -  0.72397 


50 


8.0  COST  ASSESSMENT 


This  section  provides  cost  information  to  aid  in  helping  professional  involved  in  MR  project  to 
reasonably  estimate  costs  for  implementation  at  a  given  site.  This  section  is  broken  down  into 
sub-sections  that  discuss  the  cost  model,  cost  drivers,  and  cost  benefit  of  the  various  technologies 
employed  at  CR  at  Fort  Bliss. 

8.1  COST  MODEL 

Arcadis  tracked  costs  throughout  the  ESTCP  live  site  demonstration  at  CR  and  developed  a  simple 
cost  model  to  aid  professionals  in  the  field  to  understand  costing  implications.  The  cost  model 
reflects  all  cost  elements  that  were  required  to  plan,  analyze  the  TEMTADS  data,  and  report  on 
the  results.  It  does  not  include  field  activities  that  were  conducted  by  another  contractor.  Table  8- 
1  presents  the  cost  elements  for  planning,  analyzing  the  TEMTADS  data,  and  reporting  Arcadis’ 
results  for  the  CR  live  site  demonstration.  Field  tasks  were  not  part  of  this  demonstration; 
therefore,  field  costs  are  not  included  on  Table  8-1. 

Table  8-1:  Details  of  the  Costs  Tracked  by  Arcadis 


Cost 

Element 

Data  Tracked  During  Demonstration 

Estimated  Costs 

Project 

Planning 

•  Develop  project-specific 
plans: 

o  Demonstration  Plan 
(doesn’t  include  accident 
prevention  plan, 
explosives  siting  plan  or 
other  plans. 

•  General  pre-planning 
activities 

$6,000 

TEMTADS 

Data 

Analysis 

Costs 

For  1,491  anomalies: 

•  Data  Processing  and  QC 

•  Target  parameter  extraction 

•  Advanced  anomaly 
classification 

Total  Cost  (1491  anomalies) 

•  Analyst  1:  $1 1,454  (83  hours) 

•  Analyst  2:  $9,817  (106  hours) 

Cost  per  Anomaly: 

•  Analyst  1:  $7.68 

•  Analyst  2:  $6.58 

Validation 

Digging 

•  Target  reacquisition 

•  Intrusive  investigation 

•  Intrusive  results  reporting 

•  Post-dig  anomaly  QC 

Total  Cost  (1,491  anomalies):  $248,512 

Cost  per  Anomaly:  $166.4  (estimate  from 
Camp  Ellis  demonstration) 

Final  Report 

•  Develop  project-specific 
final  report 

$20,000  (estimated) 
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8.2  COST  DRIVERS 


In  general,  the  intrusive  investigation  costs  are  the  largest  cost  drivers  on  MR  projects.  Additional 
cost  drivers  include  the  following. 

•  Static  data  collection:  TEMTADS  data  collection  and  data  processing  requires  the 
reacquisition  of  target  locations  prior  to  cued  data  collection;  therefore,  it  is  generally  more 
expensive  to  collect  data  than  with  the  MetalMapper. 

•  Intrusive  investigation  cost  savings:  The  cost  savings  associated  with  a  reduced  number 
of  non-TOI  can  lead  to  a  large  cost  savings  since  the  intrusive  investigation  costs  are  the 
largest  cost  drivers.  This  demonstration  was  limited  to  data  processing  and  analysis  and 
therefore,  Arcadis  doesn’t  have  the  exact  costs  associated  with  TEMTADS  data  collection 
and  intrusive  investigation  for  this  particular  site.  However,  Arcadis  assumes  that 
performing  AGC  would  have  been  more  cost  effective  than  intrusively  investigating  all 
targets  at  CR  if  the  background  data  quality  was  sufficient  for  performing  AGC  and  the 
Arcadis  analysts  correctly  classified  all  targets. 

8.3  COST  BENEFIT 

The  primary  driver  for  implementing  advanced  classification  is  to  reduce  the  number  of  non-TOI 
targets  that  require  intrusive  investigation  and  thereby,  decrease  the  overall  costs  of  DoD’s  MMRP 
cost  to  complete.  AGC  has  been  shown  to  reduce  the  overall  number  of  non-TOI  digs  by  60-90% 
at  other  sites.  Arcadis  successfully  reduced  the  number  of  non-TOI  digs  by  approximately  70%  at 
Pole  Mountain,  while  retaining  all  the  TOI.  However,  both  Arcadis  analysts  were  unable  to 
correctly  classify  all  the  TOI  at  CR;  therefore,  we  were  unsuccessful  at  significantly  reducing  the 
number  of  clutter  digs  while  recovering  all  the  TOI.  This  was  a  function  of  the  poor  background 
data  quality,  as  well  as  the  missed  TOI  (partial  37-mm  projectiles  and  one  2 5 -mm  projectile)  being 
most  similar  in  size  and  shape  to  20-mm  projectiles  and  neither  analyst  including  20-mm 
projectiles  in  the  TOI  list. 
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9.0  IMPLEMENTATION  ISSUES 


Arcadis  didn’t  collect  the  TEMTADS  data  at  the  CR  MRS  at  Fort  Bliss;  therefore,  we  did  not 
encounter  implementation  issues  associated  with  data  collection.  However,  based  on  our  analysis 
of  the  background  data  collected  by  URS,  it’s  apparent  that  there  was  significant  background 
variation  at  individual  background  locations.  This  could  have  been  due  to  there  being  metallic 
objects  close  to  the  background  location  and/or  the  TEMTADS  not  being  placed  back  on  the 
respective  background  locations  in  a  consistent  manner  (e.g.,  could  have  been  slightly  off  in 
location  and/or  orientation).  Due  to  the  background  variation,  each  analyst  found  it  challenging  to 
determine  the  trade-off  point  between  removing  background  data  and  keeping  enough  background 
measurements  to  level  the  data.  Each  analyst  removed  a  significant  number  of  background  data 
prior  to  leveling  cued  data;  however,  each  then  needed  to  level  cued  data  with  background  data 
points  that  were  up  to  6  hours  from  the  cued  data.  More  rigorous  background  QC  procedures  have 
been  developed  and  demonstrated  on  other  sites,  such  as  Arcadis’  Twentynine  Palms 
Demonstration  (Stacy,  2015).  These  procedures  should  continue  to  be  used  and  updated  as  needed 
to  ensure  that  high  quality  background  data  is  collected. 

In  addition,  the  ESTCP  program  office  informed  Arcadis  that  20-mm  projectiles  were  not  a  TOI 
on  the  CR  live  site;  therefore,  both  analysts  removed  20-mm  projectiles  from  the  TOI  library  to 
minimize  the  number  of  small  clutter  that  needed  to  be  dug.  No  20-mm  projectiles  were  found, 
however,  the  TOI  missed  by  both  analysts  included  a  25-mm  projectile  and  partial  37-mm 
projectiles  that  best  fit  to  20-mm  projectiles.  In  future  demonstrations  or  on  production  sites,  the 
data  analysts  should  request  further  information  about  whether  partial  projectiles  that  may  not 
contain  an  explosive  hazard  are  considered  TOI  and  perform  additional  analyses  to  ensure  that 
small  non-hazardous  MD  items  considered  TOI  are  recovered. 
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Appendix  A:  Points  of  Contact 


QAPP 

Recipients 

Title 

Organization 

Telephone 

Number 

(optional) 

E-mail  Address  or  Mailing 
Address 

Dr.  Herb 
Nelson 

Executive  Director 
and  MR  Program 
Manager 

ESTCP 

571-372- 

6400 

ESTCP  Program  Office 

4800  Mark  Center  Drive, 
Suite  17D08,  Alexandria,  VA 
22350-3605 

Herbert.h.nelson  1 0.  civ©,mail.mil 

Ms. 

Kristen 

Lau 

MR  Technical 
Assistant 

Noblis,  Inc. 

703-610- 

1622 

Noblis 

3150  Fairview  Park  Drive 
South 

Falls  Church,  VA  22042 
Kristen.Lau(S),Noblis.org 

Mr.  Steve 
Stacy 

Principal 

Investigator/Principal 
Geophysicist  (Data 
Analyst  1) 

Arcadis  ELS., 
Inc. 

703-465- 

4234 

Arcadis  U.S.,  Inc. 

2101  L.  St.  NW,  Suite  200 
Washington,  DC  20037 
Steve.  Stacv(S),arcadis. com 

Mr. 

Gabriel 

Hebert 

Project  Geophysicist 
(Data  Analyst  2) 

Arcadis  U.S., 
Inc. 

317-236- 

2859 

Arcadis  U.S.,  Inc. 

132  E.  Washington  Street, 
Suite  600 

Indianapolis,  IN  46204 
Gabriel.Hebert(a),arcadis.com 
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MEMO 


^ARCADIS  SES8* *"' 


SUBJECT:  FORT  BLISS  ESTCP  ANOMALY  DECISION  MEMO  TO 

Dr.  Herb  Nelson 

DATE  OUR  REF 

13  October,  2015 

PROJECT  NUMBER 

FROM  04659010 

Steve  Stacy 


DEPARTMENT 

ARCADIS 


COPIES  TO 

Not  Applicable 


This  anomaly  decision  memo  documents  the  procedures  that  Arcadis  Analyst  1,  Steve  Stacy,  used  to 
develop  a  ranked  anomaly  list  for  the  Advanced  Geophysical  Classification  Live  Site  Demonstration  on  the 
Former  Castner  Range  at  Fort  Bliss.  Anomalies  were  classified  using  the  procedures  outlined  in  SOP  MEC- 
AC-12  that  was  included  as  an  appendix  to  the  demonstration  plan.  Each  target  was  analysed  using  the 
single  source  and  multi-object  solver  algorithms  in  the  UX-Analyze  advanced  module  within  Oasis  montaj. 
Both  intrinsic  (size,  shape,  materials  properties)  and  extrinsic  (location,  depth,  orientation)  parameters  were 
estimated  in  these  analyses  and  a  list  of  the  relevant  target  parameters  from  each  analysis  compiled. 

ARCADIS  Analyst  1  requested  two  sets  of  training  data.  The  first  training  data  request  included  43  targets 
and  the  second  training  request  included  7  targets.  After  the  training  data  was  requested,  Arcadis  Analyst 
1  refined  the  TOI  library  and  classified  each  target  to  produce  a  ranked  target  list. 

Anomalies  were  classified  into  one  of  the  following  categories,  and  a  dig/no-dig  decision  was  made  on  each 
target: 

•  Category  -1 :  Training  Data 

•  Category  0:  Can’t  Analyze 

•  Category  1 :  TOI  (Highly  likely  to  be  MEC); 

•  Category  2:  Unsure  whether  the  TOI  is  TOI  or  non-TOI;  and 

•  Category  3:  Non-TOI  (Highly  unlikely  to  be  MEC). 

Table  1  presents  the  values  that  were  used  to  prioritize  and  rank  anomalies.  The  default  values  were  used 
for  all  inputs,  except  that  the  minimum  size  value  was  changed  to  -5  to  avoid  potentially  incorrectly 
classifying  small  TOI. 

The  decision  statistic  within  UX-Analyze  was  used  to  classify  targets  as  Category  1  through  3  as  follows: 

•  Category  1  targets  are  those  with  decision  statistics  greater  than  0.925. 

•  Category  2  targets  were  defined  as  follows: 

o  Targets  that  met  all  criteria  in  Table  1  with  decision  statistics  between  0.825  and  0.925, 
o  Non-TOI  weak  targets  that  had  decision  statistics  above  0.75,  and 

•  Category  3  targets  are  those  with  decision  statistics  below  0.825. 
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Table  1:  Advanced  Geophysical  Classification  Prioritization  and  Ranking  Inputs 


Thresholds 

Minimum 

Maximum 

Signal  Amplitude 

2 

100 

Fit  Depth 

NA 

2 

Decay 

0 

0.138 

Size 

-5 

3.144 

Diff  -  array  position  &lnverted 

Location 

NA 

0.4 

Diff  -  array  position 

&  flag  location 

NA 

0.75 

Diff  -  Flag  Location  &  inverted 

location 

NA 

0.6 

Fit  coherence 

0.8 

NA 

Boundary  of  buffer  and  TOI 

NA 

0.925 

Boundary  of  buffer  and  non- 

TOI 

NA 

0.825 

Boundary  of  buffer  and  non- 

TOI  weak  targets 

NA 

0.75 

FHigh  confidence  match  to 

clutter 

NA 

0.925 

Can’t  Analyze  anomalies  were  determined  based  on  one  or  more  of  the  following  criteria: 

•  There  are  no  polarizabilities  because  the  inversion  did  not  finish; 

•  The  fit  depth  is  unreasonable  (e.g,.  greater  than  2  meter  [mj  below  ground  surface  [bgs]) 

•  There  are  negative  polarizabilities; 

•  There  is  a  poor  fit  coherence  (e.g.,  less  than  0.8);  and 

•  Reliable  polarizabilities  couldn’t  be  extracted  (e.g.,  array  center  to  fit  location  greater  than  0.4m). 

The  attached  ranked  anomaly  list  is  ordered  from  Category  -1  through  Category  3.  The  first  items  on  the 
ranked  anomaly  list  are  the  targets  for  which  ground  truth  labels  have  been  requested  (training  data). 
Following  this,  anomalies  for  which  reliable  parameters  cannot  be  extracted  and  therefore  must  be  dug  are 
listed.  Next  are  the  items  that  Arcadis  Analyst  1  is  the  most  confident  are  “highly  likely”  to  be  TOI.  The 
items  were  ranked  according  to  decreasing  confidence  that  the  item  is  TOI  (i.e.,  decreasing  decision 
statistic).  Any  items  that  Analyst  1  was  able  to  analyze,  but  is  not  able  to  classify  (i.e.,  category  2  anomalies) 
are  placed  next  on  the  anomaly  list.  Finally,  all  those  items  that  Analyst  1  is  confident  are  not  TOI  were 
classified  as  Category  3  anomalies  and  ranked  by  their  decreasing  decision  statistic. 

In  addition  to  the  ranking,  Analyst  1  provided  an  assignment  to  the  ‘Type’  column,  indicating  the  relative 
size  of  the  type  of  munition  per  ESTCP  program  office  requirements. 

The  stop  dig  threshold  is  the  boundary  between  Category  2  and  3  anomalies  and  was  set  at  a  decision 
statistic  of  0.825  for  targets  that  met  all  criteria  and  0.75  for  non-TOI  weak  targets.  All  anomalies  with  a 
Category  -1,  0,  1,  and  2  ranking  were  placed  above  the  stop-dig  threshold  and  all  Category  3  anomalies 
were  placed  below  the  stop  dig  threshold. 

Data  was  collected  more  than  once  for  numerous  anomaly  locations.  Because  only  one  decision  is  required 
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for  each  anomaly,  Analyst  1  included  only  the  ranking  that  corresponds  to  the  most  hazardous  item  (i.e., 
the  item  with  the  highest  decision  statistic). 

If  requested,  Analyst  1  can  provide  the  preliminary  database  of  anomaly  classifications  to  the  ESTCP 
program  office. 

If  you  have  any  questions,  please  contact  Mr.  Stacy  at  703-465-4234  or  via  e-mail  at 
steve.stacy@arcadis.com. 

Thank  you, 

Steve  Stacy 
Principal  Geophysicist 

Enc.  FtBliss  ARCADIS1  LibMatch  None  2x2  None  si  vl.csv 
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MEMO 


^ARCADIS 


Design  &  Consultancy 
for  natural  and 
built  assets 


SUBJECT:  FORT  BLISS  ESTCP  -  MISSED  QC  SEED  FAILURE 
FOR  ANOMALY  CR-0780 


TO 

Dr.  Herb  Nelson 


OUR  REF 


DATE 

21  October,  2015 


PROJECT  NUMBER 

04659010 


FROM 

Gabriel  Hebert 


DEPARTMENT 

ARCADIS 


COPIES  TO 

Not  Applicable 


Introduction 

This  technical  memorandum  serves  as  a  Failure  Analysis  for  the  TEMTADS  2x2  (TEM)  dig  list  submitted  by 
Arcadis  Analyst  2  (Gabriel  Hebert)  to  the  Environmental  Securities  Technology  Certification  Program 
(ESTCP)  for  the  Fort  Bliss  Site.  Arcadis  Analyst  2  initially  submitted  a  dig  list  to  ESTCP  using  the  procedures 
outlined  in  SOP  MEC-AC-1 2  that  was  included  as  an  appendix  to  the  demonstration  plan.  Each  target  was 
analysed  using  the  single  source  and  multi-object  solver  algorithms  in  the  UX-Analyze  advanced  module 
within  Oasis  montaj.  Both  intrinsic  (size,  shape,  materials  properties)  and  extrinsic  (location,  depth, 
orientation)  parameters  were  estimated  in  these  analyses  and  a  list  of  the  relevant  target  parameters  from 
each  analysis  compiled. 

Prior  to  submitting  the  dig  list,  Arcadis  Analyst  2  requested  two  sets  of  training  data.  The  first  training  data 
request  included  25  targets  and  the  second  training  request  included  3  targets.  After  the  training  data  was 
requested,  Arcadis  Analyst  2  refined  the  TOI  library  and  classified  each  target  to  produce  a  ranked  target 
list.  However,  after  the  TEM  dig  list  was  submitted,  Arcadis  Analyst  2  was  informed  that  a  failure  memo 
containing  a  root  cause  analysis  and  a  revised  dig  list  should  be  generated,  due  to  a  failure  to  correctly  mark 
a  seed  item  (CR-0780)  for  intrusive  investigation.  This  memo  documents  the  failure  analysis  and  the 
changes  made  to  the  TEM  dig  list  that  is  attached  to  this  memo. 

Failure  Analysis 

ESTCP  informed  Arcadis  Analyst  2  that  anomaly  CR-0780  included  one  of  the  small  industry  standard  object 
(ISO)  quality  control  (QC)  seeds  buried  at  the  site,  and  that  it  was  incorrectly  not  marked  for  excavation. 
Figure  1  shows  the  dig  results  for  CR-0780.  In  addition  to  the  seed  item,  a  M48  series  fuse  component 
was  also  identified  at  the  location  of  CR-0780  (see  Figure  1). 

For  reference,  in  the  original  TEM  dig  list  (FtBliss_ARCADIS2_LIBMatch_None_2x2_None_s1_v1  .csv)  the 
decision  statistics  within  UX-Analyze  was  used  to  classify  targets  as  Category  1  through  3  as  follows: 

•  Category  1  targets  are  those  with  decision  statistics  greater  than  0.925. 

•  Category  2  targets  are  those  with  decision  statistics  between  0.925  and  0.71 . 

•  Category  3  targets  are  those  with  decision  statistics  below  0.71 . 

Can’t  Analyze  anomalies  were  determined  based  on  one  or  more  of  the  following  criteria: 

•  There  are  no  polarizabilities  because  the  inversion  did  not  finish; 

•  The  fit  depth  is  unreasonable  (e.g,.  greater  than  2  meter  [m]  below  ground  surface  [bgs]) 
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•  There  are  negative  polarizabilities; 

•  There  is  a  poor  fit  coherence  (e.g.,  less  than  0.8);  and 

•  Reliable  polarizabilities  couldn’t  be  extracted  (e.g.,  array  center  to  fit  location  greater  than  0.4m). 

In  the  original  TEM  dig  list,  target  CR-0780  was  assigned  a  decision  statistic  of  0.63,  and  was  as  such, 
classified  as  a  Category  3  target.  Aside  from  the  obvious  reason  for  misidentifying  the  seed  item  present  at 
target  location  CR-0780  (a  significant  piece  of  munitions  debris  located  in  immediate  proximity  to  the  seed 
item),  upon  further  inspection,  it  was  determined  that  target  CR-0780  was  initially  corrected  using  a  noisier 
than  average  background  location.  Additionally,  during  the  root  cause  analysis  and  background  inspection 
process,  two  additional  “bad"  background  locations  were  identified.  In  total,  1 65  targets  had  been  processed 
using  the  three  background  locations  in  question.  As  such,  these  165  targets  were  re-processed  with  the 
three  background  locations  removed  from  the  list  background  readings.  After  this  process,  CR-0780  was 
assigned  a  revised  decision  statistic  of  0.79,  which  now  falls  well  within  the  boundary  of  Category  2  targets 
(decision  statistic  between  0.92  and  0.71 ).  At  this  point,  the  re-processed  targets  (including  CR-0780)  were 
merged  with  original  targets  and  then  re-classified  and  ranked. 

Conclusions 

The  failure  to  correctly  classify  anomaly  CR-0780  as  a  TOI  was  due  to  improper  processing  due  to  a  bad 
background  location,  and  the  adjacent  presence  of  a  significant  piece  of  munitions  debris,  both  of  which  led 
to  a  poor  match  to  known  TOI  within  the  library. 


If  you  have  any  questions,  please  contact  Mr.  Hebert  at  317-694-3727  or  via  e-mail  at 
qabriel.hebert@arcadis.com. 

Thank  you, 

Gabriel  Hebert 

Project  Geophysical  Specialist 


C:\Users\M30332\AppData\Local\Temp\wze263\B  -  Ranked  Anomaly  Lists  and  Anomaly  Selection  Memos\ARCADIS2  ESTCP  Fort  Bliss  TEM  Anomaly  Failure 
Memo.docxC:\Users\M30332\AppData\Local\Temp\wze263\B  -  Ranked  Anomaly  Lists  and  Anomaly  Selection  Memos\ARCADIS2  ESTCP  Fort  Bliss  TEM  Anomaly  Failure 
Memo.docx 


2 


Figure  1:  Anomaly  CR-0780  Dig  Results 
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ESTCP  Classification  Demonstration  Ranked  Anomaly  List 

Site  FtBliss 

Analyst  Arcadisl 

Method  LibMatch 

Dynamic  D,  None 

Cued  Data  2x2 

Training  Se  Custom 

Stage  2 

Version  2 

List  all  anomalies  in  both  Training  and  Test  Sets: 

Include  prefixes  in  Target  ID  column  (for  example:  BE-138).  Data  in  all  other  columns  should  be  numerical  (no  text 

Target  ID: 

Category:  Dig 

TOI  Size 

CR-1246 

-1 

1 

CR-0046 

-1 

1 

CR-0069 

-1 

1 

CR-0110 

-1 

1 

CR-0140 

-1 

1 

CR-0218 

-1 

1 

CR-0242 

-1 

1 

CR-0244 

-1 

1 

CR-0444 

-1 

1 

CR-0453 

-1 

1 

CR-0459 

-1 

1 

CR-0474 

-1 

1 

CR-0520 

-1 

1 

CR-0546 

-1 

1 

CR-0560 

-1 

1 

CR-0589 

-1 

1 

CR-0613 

-1 

1 

CR-0639 

-1 

1 

CR-0657 

-1 

1 

CR-0686 

-1 

1 

CR-0757 

-1 

1 

CR-0780 

-1 

1 

CR-0811 

-1 

1 

CR-0901 

-1 

1 

CR-0904 

-1 

1 

CR-0915 

-1 

1 

CR-0923 

-1 

1 

CR-0970 

-1 

1 

CR-0977 

-1 

1 

CR-0993 

-1 

1 

CR-0995 

-1 

1 

CR-1011 

-1 

1 

CR-1059 

-1 

1 

CR-1120 

-1 

1 

CR-1160  -1 

CR-1200  -1 

CR-1212  -1 

CR-1223  -1 

CR-1254  -1 

CR-1271  -1 

CR-1284  -1 

CR-1306  -1 

CR-1323  -1 

CR-1324  -1 

CR-1347  -1 

CR-1365  -1 

CR-1396  -1 

CR-1417  -1 

CR-1418  -1 

CR-1421  -1 

CR-1492  -1 

CR-0023  0 

CR-0025  0 

CR-0033  0 

CR-0043  0 

CR-0052  0 

CR-0054  0 

CR-0083  0 

CR-0106  0 

CR-0113  0 

CR-0114  0 

CR-0117  0 

CR-0132  0 

CR-0162  0 

CR-0190  0 

CR-0191  0 

CR-0199  0 

CR-0207  0 

CR-0256  0 

CR-0277  0 

CR-0329  0 

CR-0331  0 

CR-0356  0 

CR-0360  0 

CR-0366  0 

CR-0404  0 

CR-0414  0 

CR-0423  0 

CR-0428  0 

CR-0452  0 

CR-0467  0 


CR-0528  0 

CR-0533  0 

CR-0609  0 

CR-0623  0 

CR-0669  0 

CR-0727  0 

CR-0729  0 

CR-0733  0 

CR-0745  0 

CR-0755  0 

CR-0787  0 

CR-0800  0 

CR-0805  0 

CR-0813  0 

CR-0816  0 

CR-0833  0 

CR-0849  0 

CR-0851  0 

CR-0854  0 

CR-0866  0 

CR-0873  0 

CR-0876  0 

CR-0891  0 

CR-0920  0 

CR-0925  0 
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Target  ID: 

Category:  Dig 

TOI  Size 

CR-0137 

-1 

1 

CR-0140 

-1 

1 

CR-0148 

-1 

1 

CR-0221 

-1 

1 

CR-0267 

-1 

1 

CR-0306 

-1 

1 

CR-0309 

-1 

1 

CR-0313 

-1 

1 

CR-0339 

-1 

1 

CR-0360 

-1 

1 

CR-0379 

-1 

1 

CR-0389 

-1 

1 

CR-0417 

-1 

1 

CR-0510 

-1 

1 

CR-0514 

-1 

1 

CR-0686 

-1 

1 

CR-0704 

-1 

1 

CR-0736 

-1 

1 

CR-0780 

-1 

1 

CR-0785 

-1 

1 

CR-0970 

-1 

1 

CR-0971 

-1 

1 

CR-1062 

-1 

1 

CR-1098 

-1 

1 

CR-1177 

-1 

1 

CR-1270 

-1 

1 

CR-1360 

-1 

1 

CR-1370 

-1 

1 

CR-1372 

-1 

1 

CR-0033 

0 

1 

CR-0034 

0 

1 

CR-0043 

0 

1 

CR-0052 

0 

1 

CR-0075 

0 

1 

CR-0088  0 

CR-0091  0 

CR-0126  0 

CR-0159  0 

CR-0197  0 

CR-0198  0 

CR-0213  0 

CR-0261  0 

CR-0262  0 

CR-0324  0 

CR-0334  0 

CR-0349  0 

CR-0355  0 

CR-0386  0 

CR-0392  0 

CR-0420  0 

CR-0485  0 

CR-0486  0 

CR-0495  0 

CR-0525  0 

CR-0533  0 

CR-0564  0 

CR-0575  0 

CR-0598  0 

CR-0615  0 

CR-0623  0 

CR-0645  0 

CR-0669  0 

CR-0700  0 

CR-0714  0 

CR-0718  0 

CR-0755  0 

CR-0803  0 

CR-0813  0 

CR-0849  0 

CR-0853  0 

CR-0859  0 

CR-0891  0 

CR-0914  0 

CR-0920  0 

CR-0956  0 

CR-1026  0 

CR-1041  0 

CR-1053  0 

CR-1066  0 

CR-1072  0 

CR-1073  0 


CR-1099  0 

CR-1118  0 

CR-1127  0 

CR-1195  0 

CR-1236  0 

CR-1251  0 

CR-1293  0 

CR-1295  0 

CR-1302  0 

CR-1305  0 

CR-1306  0 

CR-1328  0 

CR-1333  0 

CR-1340  0 

CR-1342  0 

CR-1344  0 

CR-1362  0 

CR-1373  0 

CR-1381  0 

CR-1385  0 

CR-1395  0 

CR-1396  0 

CR-1402  0 

CR-1410  0 

CR-1420  0 

CR-1451  0 

CR-1469  0 

CR-0681  1 

CR-1130  1 

CR-1142  1 

CR-0047  1 

CR-0069  1 

CR-0272  1 

CR-0375  1 

CR-0070  1 

CR-1105  1 

CR-0038  1 

CR-1482  1 

CR-0469  1 

CR-0269  1 

CR-0172  1 

CR-0243  1 

CR-1209  1 

CR-0254  1 

CR-1422  1 

CR-0909  1 

CR-0529  1 


CR-0606 

CR-1331 

CR-0251 

CR-1015 

CR-0584 

CR-0508 

CR-0460 

CR-1284 

CR-0504 

CR-1307 

CR-1063 

CR-1016 

CR-1471 

CR-0446 

CR-0561 

CR-0436 

CR-1298 

CR-0545 

CR-1297 

CR-0482 

CR-0581 

CR-1022 

CR-0520 

CR-0372 

CR-0394 

CR-0527 

CR-0879 

CR-0365 

CR-1341 

CR-1369 

CR-1388 

CR-1336 

CR-1024 

CR-1379 

CR-1398 

CR-0582 

CR-0412 

CR-0578 

CR-0931 

CR-0440 

CR-0314 

CR-0422 

CR-0580 

CR-0761 

CR-0904 

CR-0977 

CR-0986 


111 
111 
112 
111 
112 
112 
111 
113 
111 
112 
111 
111 
112 
111 
111 
111 
111 
111 
111 
111 
111 
111 
112 
111 
111 
111 
112 
2  11 

2  11 

2  11 

2  12 

2  11 

2  12 

2  12 

2  12 

2  11 

2  11 

2  11 

2  11 

2  11 

2  12 

2  12 

2  11 

2  11 

2  11 

2  11 

2  12 


CR-1160 

CR-0734 

CR-0399 

CR-0450 

CR-0273 

CR-0757 

CR-0569 

CR-0823 

CR-0572 

CR-0733 

CR-1029 

CR-0409 

CR-0252 

CR-0146 

CR-0490 

CR-1338 

CR-0468 

CR-0505 

CR-1198 

CR-0614 

CR-1019 

CR-0444 

CR-0980 

CR-0542 

CR-0763 

CR-1491 

CR-0995 

CR-1221 

CR-0393 

CR-1052 

CR-1008 

CR-0499 

CR-1351 

CR-0538 

CR-0765 

CR-1309 

CR-0220 

CR-0984 

CR-0493 

CR-0474 

CR-0465 

CR-0065 

CR-1444 

CR-0265 

CR-0557 

CR-0847 

CR-0458 


CR-0982 

CR-0811 

CR-1268 

CR-1262 

CR-0398 

CR-0612 

CR-0929 

CR-1366 

CR-1279 

CR-0626 

CR-0459 

CR-0475 

CR-1078 

CR-0501 

CR-0768 

CR-1289 

CR-0109 

CR-0388 

CR-1060 

CR-1436 

CR-0923 

CR-0731 

CR-1403 

CR-0517 

CR-1316 

CR-1023 

CR-0445 

CR-0913 

CR-0523 

CR-1278 

CR-1039 

CR-0979 

CR-0549 

CR-0947 

CR-1020 

CR-0435 

CR-0550 

CR-0534 

CR-0516 

CR-1404 

CR-1051 

CR-1365 

CR-0340 

CR-0997 

CR-0771 

CR-0890 

CR-1384 


2  12 
2  11 
2  12 
2  11 
2  12 
2  11 
2  12 
2  11 
2  11 
2  12 
2  11 
2  11 
2  12 
2  11 
2  11 
2  12 
2  12 
2  12 
2  11 
2  12 
2  11 
2  11 
2  11 
2  12 
2  11 
2  12 
2  11 
2  11 
2  12 
2  12 
2  11 
2  12 
2  12 
2  11 
2  12 
2  12 
2  11 
2  12 
2  12 
2  12 
2  11 
2  12 
2  12 
2  12 
2  11 
2  11 
2  12 


CR-0573 

CR-1441 

CR-0985 

CR-0387 

CR-0742 

CR-0437 

CR-1303 

CR-0605 

CR-0513 

CR-0917 

CR-0769 

CR-1291 

CR-1290 

CR-1485 

CR-1467 

CR-0791 

CR-1389 

CR-0961 

CR-1363 

CR-0498 

CR-0410 

CR-0518 

CR-0754 

CR-0822 

CR-1276 

CR-0933 

CR-1082 

CR-0602 

CR-0756 

CR-1238 

CR-1077 

CR-1312 

CR-0500 

CR-0730 

CR-0932 

CR-0537 

CR-1046 

CR-0903 

CR-0434 

CR-1354 

CR-0411 

CR-0509 

CR-1319 

CR-0576 

CR-0748 

CR-0976 

CR-1466 


2  12 
2  11 
2  11 
2  11 
2  12 
2  12 
2  11 
2  11 
2  12 
2  11 
2  11 
2  11 
2  12 
2  11 
2  11 
2  12 
2  12 
2  12 
2  11 
2  11 
2  12 
2  11 
2  12 
2  12 
2  11 
2  12 
2  12 
2  11 
2  12 
2  11 
2  11 
2  11 
2  12 
2  11 
2  11 
2  12 
2  12 
2  11 
2  11 
2  12 
2  11 
2  11 
2  12 
2  11 
2  12 
2  12 
2  11 


CR-0599 

CR-0579 

CR-0772 

CR-1009 

CR-0466 

CR-1294 

CR-0521 

CR-0912 

CR-0439 

CR-1094 

CR-0400 

CR-1376 

CR-0035 

CR-1038 

CR-0280 

CR-0149 

CR-1456 

CR-0770 

CR-0524 

CR-0244 

CR-1042 

CR-0060 

CR-1401 

CR-0030 

CR-0739 

CR-0583 

CR-1246 

CR-0451 

CR-0993 

CR-0759 

CR-0783 

CR-0753 

CR-1393 

CR-0740 

CR-1430 

CR-0275 

CR-1138 

CR-1043 

CR-0304 

CR-0384 

CR-1254 

CR-1375 

CR-0032 

CR-0764 

CR-0894 

CR-0708 

CR-0021 


2  11 
2  11 
2  11 
2  11 
2  11 
2  12 
2  11 
2  12 
2  11 
2  11 
2  11 
2  12 
2  11 
2  11 
2  12 
2  11 
2  11 
2  12 
2  12 
2  12 
2  12 
2  11 
2  11 
2  11 
2  11 
2  11 
2  11 
2  11 
2  11 
2  12 
2  12 
2  11 
2  11 
2  11 
2  11 
2  11 
2  11 
2  11 
2  11 
2  11 
2  12 
2  11 
2  12 
2  11 
2  11 
2  11 
2  11 


CR-1356 

2 

CR-1423 

2 

CR-1445 

2 

CR-1021 

2 

CR-0577 

2 

CR-0433 

2 

CR-0966 

2 

CR-1448 

2 

CR-1418 

2 

CR-0494 

2 

CR-0555 

2 

CR-1359 

2 

CR-0270 

2 

CR-0595 

2 

CR-0868 

2 

CR-1468 

2 

CR-0407 

2 

CR-0359 

2 

CR-0227 

2 

CR-0825 

2 

CR-0574 

2 

CR-0743 

2 

CR-0539 

2 

CR-1004 

2 

CR-0732 

2 

CR-0145 

2 

CR-0613 

2 

CR-1493 

2 

CR-1488 

2 

CR-0022 

2 

CR-1335 

2 

CR-1397 

2 

CR-1271 

2 

CR-1005 

3 

CR-1012 

3 

CR-1492 

3 

CR-0989 

3 

CR-1007 

3 

CR-1390 

3 

CR-0796 

3 

CR-1108 

3 

CR-1000 

3 

CR-1408 

3 

CR-1431 

3 

CR-1486 

3 

CR-0727 

3 

CR-1100 

3 

CR-0762 

3 

0 

CR-0639 

3 

0 

CR-0870 

3 

0 

CR-0798 

3 

0 

CR-1234 

3 

0 

CR-1364 

3 

0 

CR-0182 

3 

0 

CR-0418 

3 

0 

CR-1215 

3 

0 

CR-1010 

3 

0 

CR-0478 

3 

0 

CR-0864 

3 

0 

CR-0506 

3 

0 

CR-1068 

3 

0 

CR-0724 

3 

0 

CR-1219 

3 

0 

CR-0461 

3 

0 

CR-0725 

3 

0 

CR-0502 

3 

0 

CR-0641 

3 

0 

CR-0511 

3 

0 

CR-0354 

3 

0 

CR-0247 

3 

0 

CR-0531 

3 

0 

CR-1255 

3 

0 

CR-0936 

3 

0 

CR-1097 

3 

0 

CR-0230 

3 

0 

CR-1202 

3 

0 

CR-0132 

3 

0 

CR-0143 

3 

0 

CR-0935 

3 

0 

CR-1135 

3 

0 

CR-1055 

3 

0 

CR-0447 

3 

0 

CR-0955 

3 

0 

CR-1199 

3 

0 

CR-1121 

3 

0 

CR-1106 

3 

0 

CR-1250 

3 

0 

CR-0526 

3 

0 

CR-0661 

3 

0 

CR-0128 

3 

0 

CR-1137 

3 

0 

CR-0857 

3 

0 

CR-0585 

3 

0 

CR-0274 

3 

0 

CR-1361 

3 

0 

CR-0570 

3 

0 

CR-0019 

3 

0 

CR-1357 

3 

0 

CR-0186 

3 

0 

CR-1269 

3 

0 

CR-0965 

3 

0 

CR-0536 

3 

0 

CR-0922 

3 

0 

CR-1047 

3 

0 

CR-0797 

3 

0 

CR-0988 

3 

0 

CR-0640 

3 

0 

CR-0954 

3 

0 

CR-0819 

3 

0 

CR-0264 

3 

0 

CR-0222 

3 

0 

CR-1475 

3 

0 

CR-0773 

3 

0 

CR-1017 

3 

0 

CR-0429 

3 

0 

CR-0129 

3 

0 

CR-1323 

3 

0 

CR-0353 

3 

0 

CR-0283 

3 

0 

CR-0901 

3 

0 

CR-1172 

3 

0 

CR-0695 

3 

0 

CR-0729 

3 

0 

CR-1187 

3 

0 

CR-0981 

3 

0 

CR-0503 

3 

0 

CR-1457 

3 

0 

CR-1233 

3 

0 

CR-1494 

3 

0 

CR-0882 

3 

0 

CR-0326 

3 

0 

CR-1059 

3 

0 

CR-1129 

3 

0 

CR-1149 

3 

0 

CR-0735 

3 

0 

CR-0029 

3 

0 

CR-1148 

3 

0 

CR-1334 

3 

0 

CR-0512 

3 

0 

CR-1414 

3 

0 

CR-1003 

3 

0 

CR-0315 

3 

0 

CR-0383 

3 

0 

CR-0594 

3 

0 

CR-1223 

3 

0 

CR-1208 

3 

0 

CR-1214 

3 

0 

CR-0760 

3 

0 

CR-0738 

3 

0 

CR-0397 

3 

0 

CR-1285 

3 

0 

CR-1435 

3 

0 

CR-1002 

3 

0 

CR-0648 

3 

0 

CR-0767 

3 

0 

CR-0996 

3 

0 

CR-0706 

3 

0 

CR-0886 

3 

0 

CR-1347 

3 

0 

CR-1438 

3 

0 

CR-0601 

3 

0 

CR-0854 

3 

0 

CR-1025 

3 

0 

CR-0945 

3 

0 

CR-0203 

3 

0 

CR-0352 

3 

0 

CR-0808 

3 

0 

CR-1028 

3 

0 

CR-0752 

3 

0 

CR-1299 

3 

0 

CR-0751 

3 

0 

CR-1144 

3 

0 

CR-0717 

3 

0 

CR-1018 

3 

0 

CR-0347 

3 

0 

CR-1045 

3 

0 

CR-0290 

3 

0 

CR-0072 

3 

0 

CR-1327 

3 

0 

CR-1164 

3 

0 

CR-0169 

3 

0 

CR-0196 

3 

0 

CR-1274 

3 

0 

CR-0155 

3 

0 

CR-0139 

3 

0 

CR-1459 

3 

0 

CR-0218 

3 

0 

CR-1447 

3 

0 

CR-1382 

3 

0 

CR-0368 

3 

0 

CR-0231 

3 

0 

CR-0005 

3 

0 

CR-0442 

3 

0 

CR-0131 

3 

0 

CR-1081 

3 

0 

CR-1096 

3 

0 

CR-0164 

3 

0 

CR-1170 

3 

0 

CR-1399 

3 

0 

CR-1391 

3 

0 

CR-1034 

3 

0 

CR-1128 

3 

0 

CR-0927 

3 

0 

CR-1487 

3 

0 

CR-1006 

3 

0 

CR-0165 

3 

0 

CR-1406 

3 

0 

CR-0593 

3 

0 

CR-1352 

3 

0 

CR-0926 

3 

0 

CR-1275 

3 

0 

CR-0064 

3 

0 

CR-1437 

3 

0 

CR-0779 

3 

0 

CR-0363 

3 

0 

CR-1117 

3 

0 

CR-0382 

3 

0 

CR-0692 

3 

0 

CR-0678 

3 

0 

CR-1383 

3 

0 

CR-1433 

3 

0 

CR-0673 

3 

0 

CR-1343 

3 

0 

CR-1392 

3 

0 

CR-1292 

3 

0 

CR-0696 

3 

0 

CR-0395 

3 

0 

CR-1317 

3 

0 

CR-1415 

3 

0 

CR-0646 

3 

0 

CR-1378 

3 

0 

CR-0983 

3 

0 

CR-0967 

3 

0 

CR-1280 

3 

0 

CR-0071 

3 

0 

CR-1175 
CR-1040 
CR-1245 
CR-0323 
CR-1259 
CR-1143 
CR-0697 
CR-0403 
CR-1049 
CR-0631 
CR-0476 
CR-0758 
CR-0634 
CR-0016 
CR-1462 
CR-0629 
CR-0650 
CR-1165 
CR-0638 
CR-0167 
CR-0329 
CR-0293 
CR-0591 
CR-0456 
CR-0287 
CR-0471 
CR-0074 
CR-0556 
CR-1113 
CR-0711 
CR-0600 
CR-0871 
CR-0036 
CR-0563 
CR-1103 
CR-0653 
CR-1180 
CR-0189 
CR-0046 
CR-0470 
CR-0837 
CR-0014 
CR-0664 
CR-0944 
CR-0799 
CR-0101 
CR-0181 


0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 


CR-1168 
CR-1377 
CR-0546 
CR-0496 
CR-0705 
CR-1442 
CR-0636 
CR-1211 
CR-1136 
CR-0001 
CR-0701 
CR-0229 
CR-0097 
CR-1169 
CR-0607 
CR-0239 
CR-0371 
CR-0467 
CR-1181 
CR-0655 
CR-0378 
CR-0156 
CR-0361 
CR-1179 
CR-0110 
CR-0911 
CR-0356 
CR-0040 
CR-0212 
CR-0842 
CR-0362 
CR-1183 
CR-1477 
CR-0675 
CR-0810 
CR-0530 
CR-0647 
CR-1470 
CR-1031 
CR-0657 
CR-0887 
CR-0818 
CR-0672 
CR-1119 
CR-0540 
CR-0428 
CR-0845 


0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 


CR-0831 

3 

0 

CR-0668 

3 

0 

CR-0219 

3 

0 

CR-0637 

3 

0 

CR-1240 

3 

0 

CR-0794 

3 

0 

CR-0568 

3 

0 

CR-0807 

3 

0 

CR-1427 

3 

0 

CR-0199 

3 

0 

CR-1123 

3 

0 

CR-1011 

3 

0 

CR-1050 

3 

0 

CR-1320 

3 

0 

CR-1122 

3 

0 

CR-0374 

3 

0 

CR-1184 

3 

0 

CR-0541 

3 

0 

CR-1162 

3 

0 

CR-0551 

3 

0 

CR-1242 

3 

0 

CR-0604 

3 

0 

CR-0481 

3 

0 

CR-0286 

3 

0 

CR-0850 

3 

0 

CR-0869 

3 

0 

CR-0141 

3 

0 

CR-0872 

3 

0 

CR-1089 

3 

0 

CR-0624 

3 

0 

CR-0390 

3 

0 

CR-0840 

3 

0 

CR-0609 

3 

0 

CR-0741 

3 

0 

CR-0774 

3 

0 

CR-0025 

3 

0 

CR-0234 

3 

0 

CR-0844 

3 

0 

CR-0666 

3 

0 

CR-0328 

3 

0 

CR-0726 

3 

0 

CR-1182 

3 

0 

CR-1131 

3 

0 

CR-0004 

3 

0 

CR-0855 

3 

0 

CR-1463 

3 

0 

CR-1086 

3 

0 

CR-1400 
CR-0707 
CR-0571 
CR-0345 
CR-1405 
CR-0719 
CR-0682 
CR-0699 
CR-0654 
CR-0713 
CR-0151 
CR-1093 
CR-0766 
CR-0918 
CR-0630 
CR-0364 
CR-0358 
CR-0427 
CR-1265 
CR-1281 
CR-1300 
CR-0380 
CR-0018 
CR-0048 
CR-0597 
CR-1461 
CR-0990 
CR-0061 
CR-0680 
CR-0790 
CR-0235 
CR-0620 
CR-0063 
CR-0408 
CR-0658 
CR-0737 
CR-0815 
CR-1153 
CR-1186 
CR-1465 
CR-0113 
CR-0170 
CR-0778 
CR-1282 
CR-0721 
CR-1443 
CR-0192 


0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 
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